A Caltech Library Service

Combining Multiplexed Ion Beam Imaging (MIBI) with Convolutional Neural Networks to accurately segment cells in human tissue

Greenwald, Noah and Keren, Leeat and Greenbaum, Shirley and Fong, Mara and Chaudhry, Gautam and Abraham, Zion and Moseley, Jackson and Van Valen, David and Angelo, Michael (2019) Combining Multiplexed Ion Beam Imaging (MIBI) with Convolutional Neural Networks to accurately segment cells in human tissue. Journal for ImmunoTherapy of Cancer, 7 (S1). Art. No. P736. ISSN 2051-1426. doi:10.1186/s40425-019-0764-0.

[img] PDF - Published Version
Creative Commons Attribution.


Use this Persistent URL to link to this item:


Background: Multiplexed imaging is a rapidly growing field that promises to substantially increase the number of proteins that can be imaged simultaneously. We have developed Multiplexed Ion Beam Imaging by Time of Flight (MIBI-TOF), which uses elemental reporters conjugated to primary antibodies that are then quantified using a time of flight mass-spectrometer. This technique allows for more than 40 distinct proteins to visualized at once in the same clinical samples. This has already yielded significant insights into the interactions and relationships between the many different immune cell populations present in the tumor microenvironment. However, one of the remaining challenges in analyzing such data is accurately determining target protein expression values for each cell in the image. This requires the precise delineation of boundaries between cells that are often tightly packed next to one another. Current methods to address this challenge largely rely on DNA intensity to make these splits, and are thus mostly limited to nuclear segmentation. Methods: We have developed a novel convolutional neural network to perform whole-cell segmentation from multiplexed imaging data. Rather than relying only on DNA signal, we use a panel of morphological markers. Our method integrates the information from these distinct proteins, allowing it to segment large cancer cells, small lymphocytes, and normal epithelium at the same time without requiring fine-tuning or manual adjustment. Results: By combining our novel imaging platform with new computational tools, we are able to achieve extremely accurate segmentation of whole cells in tissue. Our approach compares favorably with many of the currently used tools for segmentation. We show that our improvements in accuracy come both from our novel imaging approach as well as algorithmic advances. We perform significantly better than traditional machine learning algorithms trained on the same dataset. Additionally, we show that our algorithm can be trained to identify cells across a range of cancer histologies and disease grades. Conclusions: We have developed a robust and accurate approach to whole-cell segmentation in human tissues. We show the superiority over this method over current state of the art algorithms. The accurate segmentation generated by our approach will enable the analysis of complex tissue architectures with highly overlapping cell types, and will help to advance our understanding of the interactions between cell types in the diseased state.

Item Type:Article
Related URLs:
URLURL TypeDescription
Keren, Leeat0000-0002-6799-6303
Van Valen, David0000-0001-7534-7621
Additional Information:© 2019 BioMed Central Ltd. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( applies to the data made available in this article, unless otherwise stated. Published 06 November 2019.
Issue or Number:S1
Record Number:CaltechAUTHORS:20191205-101747645
Persistent URL:
Official Citation:34th Annual Meeting & Pre-Conference Programs of the Society for Immunotherapy of Cancer (SITC 2019): part 2. j. immunotherapy cancer 7, 283 (2019) doi:10.1186/s40425-019-0764-0
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:100205
Deposited By: Tony Diaz
Deposited On:05 Dec 2019 18:32
Last Modified:16 Nov 2021 17:52

Repository Staff Only: item control page