A Caltech Library Service

Spectral organ fingerprints for intraoperative tissue classification with hyperspectral imaging

Studier-Fischer, A. and Seidlitz, S. and Sellner, J. and Wiesenfarth, M. and Ayala, L. and Özdemir, B. and Odenthal, J. and Knödler, S. and Kowalewski, K. F. and Haney, C. M. and Camplisson, I. and Dietrich, M. and Schmidt, K. and Salg, G. A. and Kenngott, H. G. and Adler, T. J. and Schreck, N. and Kopp-Schneider, A. and Maier-Hein, K. and Maier-Hein, L. and Müller-Stich, B. P. and Nickel, F. (2021) Spectral organ fingerprints for intraoperative tissue classification with hyperspectral imaging. . (Unpublished)

[img] PDF - Submitted Version
Creative Commons Attribution Non-commercial No Derivatives.

[img] PDF - Supplemental Material
Creative Commons Attribution Non-commercial No Derivatives.


Use this Persistent URL to link to this item:


Visual discrimination of tissue during surgery can be challenging since different tissues appear similar to the human eye. Hyperspectral imaging (HSI) removes this limitation by associating each pixel with high-dimensional spectral information. While previous work has shown its general potential to discriminate tissue, clinical translation has been limited due to the method’s current lack of robustness and generalizability. Specifically, it had been unknown whether variability in spectral reflectance is primarily explained by tissue type rather than the recorded individual or specific acquisition conditions. The contribution of this work is threefold: (1) Based on an annotated medical HSI data set (9,059 images from 46 pigs), we present a tissue atlas featuring spectral fingerprints of 20 different porcine organs and tissue types. (2) Using the principle of mixed model analysis, we show that the greatest source of variability related to HSI images is the organ under observation. (3) We show that HSI-based fully-automatic tissue differentiation of 20 organ classes with deep neural networks is possible with high accuracy (> 95 %). We conclude from our study that automatic tissue discrimination based on HSI data is feasible and could thus aid in intraoperative decision making and pave the way for context-aware computer-assisted surgery systems and autonomous robotics.

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Studier-Fischer, A.0000-0001-8682-9300
Seidlitz, S.0000-0002-1122-4793
Sellner, J.0000-0003-4469-8343
Wiesenfarth, M.0000-0001-6024-542X
Ayala, L.0000-0002-3574-2085
Özdemir, B.0000-0001-8927-4830
Odenthal, J.0000-0003-3612-7401
Knödler, S.0000-0001-5798-8003
Kowalewski, K. F.0000-0003-2931-6247
Haney, C. M.0000-0002-4209-9470
Camplisson, I.0000-0001-9653-2789
Dietrich, M.0000-0003-0960-038X
Schmidt, K.0000-0001-8373-9406
Salg, G. A.0000-0002-3964-3527
Kenngott, H. G.0000-0003-1123-346X
Adler, T. J.0000-0002-3424-6629
Schreck, N.0000-0001-9277-4297
Kopp-Schneider, A.0000-0002-1810-0267
Maier-Hein, K.0000-0002-6626-2463
Maier-Hein, L.0000-0003-4910-9368
Müller-Stich, B. P.0000-0002-8552-8538
Nickel, F.0000-0001-6066-8238
Additional Information:The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license. This version posted November 25, 2021. The authors thank Minu Tizabi (CAMI, DKFZ) for proofreading the manuscript and gratefully acknowledge the data storage service SDS@hd supported by the Ministry of Science, Research and the Arts Baden-Württemberg (MWK) and the German Research Foundation (DFG) through grant INST 35/1314-1 FUGG and INST 35/1503-1 FUGG. The present contribution is supported by the Helmholtz Association under the joint research school HIDSS4Health (Helmholtz Information and Data Science School for Health). This project received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation program (NEURAL SPICING, grant agreement No. [101002198]) as well as from the Willi Robert Pitzer foundation, from the Heidelberg Foundation for Surgery and from the RISE program by the German Academic Exchange Service (DAAD). Author Contributions: ASF, FN and BPMS had the original idea for the project. ASF and FN initiated the project. ASF, FN and CMH performed the initial review of existing literature and the planning. ASF, CMH, KK, BÖ, MD, BPMS and FN performed the surgeries. ASF, IC, BÖ and JO developed the Python codes for data organization and annotation. ASF, GS and SK annotated data. ASF, GS, BÖ, JO, SK, SS, LA, JS, LMH, TJA and MW analyzed and interpreted data. LMH, TJA, LA, SS, JS, MW, NS, AKS and ASF developed and implemented the statistical and machine learning-based data analysis strategy. LA provided the t-SNE analysis and the relevant manuscript passages. MW, NS and AKS provided the mixed model analysis, the structured model and the relevant manuscript passages. SS and JS provided the machine learning-based classification and the relevant manuscript passages. FN, LMH, HGK, KMH, KS and BPMS provided expert knowledge throughout the project. ASF, BÖ and FN wrote the manuscript. SS, JS, LA, MW, BPMS, HGK, KMH and LMH revised the manuscript. All authors have read and approved the final manuscript. Authors state no conflict of interest.
Funding AgencyGrant Number
Ministry of Science, Research and the Arts Baden-Wurttemberg (MWK)UNSPECIFIED
Deutsche Forschungsgemeinschaft (DFG)INST 35/1314-1 FUGG
Deutsche Forschungsgemeinschaft (DFG)INST 35/1503-1 FUGG
Helmholtz AssociationUNSPECIFIED
European Research Council (ERC)101002198
Willi Robert Pitzer FoundationUNSPECIFIED
Heidelberg Foundation for SurgeryUNSPECIFIED
Deutscher Akademischer Austauschdienst (DAAD)UNSPECIFIED
Subject Keywords:hyperspectral imaging, multispectral imaging, organ classification, tissue classification, organ identification, tissue identification, translational research, porcine model, machine learning, deep learning, linear mixed model, surgery, surgical data science
Record Number:CaltechAUTHORS:20211130-152255247
Persistent URL:
Official Citation:Spectral organ fingerprints for intraoperative tissue classification with hyperspectral imaging. A. Studier-Fischer, S. Seidlitz, J. Sellner, M. Wiesenfarth, L. Ayala, B. Özdemir, J. Odenthal, S. Knödler, K.F. Kowalewski, C.M. Haney, I. Camplisson, M. Dietrich, K. Schmidt, G.A. Salg, H.G. Kenngott, T.J. Adler, N. Schreck, A. Kopp-Schneider, K. Maier-Hein, L. Maier-Hein, B.P. Müller-Stich, F. Nickel. bioRxiv 2021.11.24.469943; doi:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:112084
Deposited By: Tony Diaz
Deposited On:30 Nov 2021 16:35
Last Modified:30 Nov 2021 16:35

Repository Staff Only: item control page