A Caltech Library Service

Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning

Moen, Erick and Borba, Enrico and Miller, Geneva and Schwartz, Morgan and Bannon, Dylan and Koe, Nora and Camplisson, Isabella and Kyme, Daniel and Pavelchek, Cole and Price, Tyler and Kudo, Takamasa and Pao, Edward and Graf, William and Van Valen, David (2019) Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning. . (Unpublished)

[img] PDF - Submitted Version
Creative Commons Attribution Non-commercial No Derivatives.


Use this Persistent URL to link to this item:


Live-cell imaging experiments have opened an exciting window into the behavior of living systems. While these experiments can produce rich data, the computational analysis of these datasets is challenging. Single-cell analysis requires that cells be accurately identified in each image and subsequently tracked over time. Increasingly, deep learning is being used to interpret microscopy image with single cell resolution. In this work, we apply deep learning to the problem of tracking single cells in live-cell imaging data. Using crowdsourcing and a human-in-the-loop approach to data annotation, we constructed a dataset of over 11,000 trajectories of cell nuclei that includes lineage information. Using this dataset, we successfully trained a deep learning model to perform cell tracking within a linear programming framework. Benchmarking tests demonstrate that our method achieves state-of-the-art performance on the task of cell tracking with respect to multiple accuracy metrics. Further, we show that our deep learning-based method generalizes to perform cell tracking for both fluorescent and brightfield images of the cell cytoplasm, despite having never been trained those data types. This enables analysis of live-cell imaging data collected across imaging modalities. A persistent cloud deployment of our cell tracker is available at

Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription Paper
Moen, Erick0000-0002-5947-7044
Schwartz, Morgan0000-0001-8131-9125
Kudo, Takamasa0000-0002-9709-5549
Van Valen, David0000-0001-7534-7621
Additional Information:The copyright holder for this preprint is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license. bioRxiv preprint first posted online Oct. 13, 2019. We thank Anima Anandkumar, Michael Angelo, Michael Elowitz, Christopher Frick, Lea Geontoro, Kerwyn Casey Huang, and Gregory Johnson for helpful suggestions and sharing data. We thank Ian Brown and Andy Butkovic for assistance using the Figure 8 image annotation platform, as well as numerous anonymous annotators whose efforts enabled this work. We also thank Henrietta Lacks for graciously donating source material. We gratefully acknowledge support from the Paul Allen Family Foundation through the Discovery Centers at Stanford University and Caltech, The Rosen Center for Bioengineering at Caltech, The Center for Environmental and Microbial Interactions at Caltech, Google Research Cloud, Figure 8’s AI for everyone award, and a subaward from NIH U24CA224309-01. Author contributions: EM, WG, and DVV conceived of the project; EM, EB, MS, DB, WG, and DVV designed and wrote the cell tracking algorithm and its deployment; EB, EM, and GM designed and wrote the Caliban software; GM designed and oversaw the data annotation; GM, NK, IC, DK, CP, and TP annotated data; MS, EM, and CP designed and performed benchmarking; TK and EP collected data for annotation; EM and DVV wrote the paper; DVV supervised the project. Datasets: All of the data used in this paper and the associated annotations can be accessed at or at through the datasets module. Source code: A persistent deployment of the software described here can be accessed at All source code for cell tracking is available in the DeepCell repository at The source code for the Caliban software is available at Detailed instructions are available at Competing interests: The authors have filed a provisional patent for the described work; the software described here is available under a modified Apache license and is free for non-commercial uses.
Group:Rosen Bioengineering Center, Caltech Center for Environmental Microbial Interactions (CEMI)
Funding AgencyGrant Number
Paul Allen Family FoundationUNSPECIFIED
Donna and Benjamin M. Rosen Bioengineering CenterUNSPECIFIED
Caltech Center for Environmental Microbial Interactions (CEMI)UNSPECIFIED
Google Research CloudUNSPECIFIED
Figure 8’s AI for EveryoneUNSPECIFIED
Record Number:CaltechAUTHORS:20191014-100546775
Persistent URL:
Official Citation:Accurate cell tracking and lineage construction in live-cell imaging experiments with deep learning. Erick Moen, Enrico Borba, Geneva Miller, Morgan Schwartz, Dylan Bannon, Nora Koe, Isabella Camplisson, Daniel Kyme, Cole Pavelchek, Tyler Price, Takamasa Kudo, Edward Pao, William Graf, David Ashley Van Valen. bioRxiv 803205; doi:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:99247
Deposited By: Tony Diaz
Deposited On:14 Oct 2019 17:28
Last Modified:16 Nov 2021 17:44

Repository Staff Only: item control page