Welcome to the new version of CaltechAUTHORS. Login is currently restricted to library staff. If you notice any issues, please email coda@library.caltech.edu
Published May 25, 2023 | In Press
Journal Article Open

The Cell Tracking Challenge: 10 years of objective benchmarking


The Cell Tracking Challenge is an ongoing benchmarking initiative that has become a reference in cell segmentation and tracking algorithm development. Here, we present a significant number of improvements introduced in the challenge since our 2017 report. These include the creation of a new segmentation-only benchmark, the enrichment of the dataset repository with new datasets that increase its diversity and complexity, and the creation of a silver standard reference corpus based on the most competitive results, which will be of particular interest for data-hungry deep learning-based strategies. Furthermore, we present the up-to-date cell segmentation and tracking leaderboards, an in-depth analysis of the relationship between the performance of the state-of-the-art methods and the properties of the datasets and annotations, and two novel, insightful studies about the generalizability and the reusability of top-performing methods. These studies provide critical practical conclusions for both developers and users of traditional and machine learning-based cell segmentation and tracking algorithms.

Additional Information

© 2023 The Author(s). This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/. The authors thank J. Padilla Pérez, who worked for many hours on the annotation of the new datasets; J.-Y. Tinevez, who kindly added the CTC measures into the popular TrackMate software and who, with T. Pietzsch, developed the Mastodon software that became instrumental for us when preparing the tracking annotations of the large embryonic datasets; and the participants of the challenge not included in the list of authors of this analysis paper, listed on the challenge website (http://celltrackingchallenge.net/participants/). This work was funded by the Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación (MCIU/AEI/10.13039/50110011033) and FEDER funds UE under Grants RTI2018-094494-B-C22, TED2021-131300B-I00, PDI2021-122409OB-C22 (C.O.S.); Czech Ministry of Education, Youth and Sports national research infrastructure Czech-BioImaging projects LM2023050 and CZ.02.1.01/0.0/0.0/18_046/0016045 (M.M., V.U. and M.K.); Czech Science Foundation (GACR) grant GA21-20374S (P.M. and F.L.); European Regional Development Fund in the IT4Innovations national super-computing center–path to exascale project CZ.02.1.01/0.0/0.0/16_013/0001791 within the Operational Programme Research, Development and Education (V.U.); Czech Ministry of Education, Youth and Sports through the e-INFRA CZ project ID:90140 (V.U.); Ministerio de Ciencia, Innovación y Universidades, Agencia Estatal de Investigación Grant PID2019-109820RB-I00, MCIN/AEI/10.13039/501100011033/, co-financed by the European Regional Development Fund (ERDF), 'A way of making Europe' (P.D.-R., E.G.M., grant to A.M.-B.); BBVA Foundation under a 2017 Leonardo Grant for Researchers and Cultural Creators (A.M.-B.). NVIDIA Corporation for the donation of the Titan X (Pascal) GPU (P.D.-R., E.G.M., to A.M.-B.); the Gulbenkian Foundation, the European Molecular Biology Organization (EMBO) Installation Grant (EMBO-2020-IG-4734) (granted to the Optical Cell Biology laboratory at Instituto Gulbenkian de Ciência) and Postdoctoral Fellowship (EMBO ALTF 174-2022) (E.G.M.); Helmholtz Association program NACIP – Natural, Artificial and Cognitive Information Processing and Biointerfaces International Graduate School (BIF-IGS) (T.S. and R.M.), and HIDSS4Health – Helmholtz Information and Data Science School for Health (K.L. and R.M.); European Research Council, under the European Union Horizon 2020 programme, grant ERC-2015-AdG 694918 (K.S.); Helmholtz Imaging (F.I., P.F.J.); the Negev scholarship at Ben-Gurion University (A.A.); the Kreitman School of Advanced Graduate Studies (A.A.); Israel Ministry of Science, Technology and Space (MOST 3-14344 T.R.R.); the United States – Israel Binational Science Foundation (BSF 2019135 T.R.R.); Human Frontiers Science Program Grant RGP0043/2019 (A.R.C. and L.A.); the Brazilian funding agencies FACEPE, CAPES and CNPq (F.A.G.P., T.I.R.); Beckman Institute at Caltech (A.C., F.A.G.P.); Howard Hughes Medical Institute (E.M.M.); and the USA NIH NINDS R01NS110915 (K.P.) and USA ARL W911NF-18-20285 (K.P.). These authors contributed equally: Michal Kozubek, Carlos Ortiz-de-Solórzano. Contributions. M.M. conceived and designed the analysis, collected the data, contributed data or analysis tools, performed the analysis and wrote the paper; V.U. conceived and designed the analysis, contributed data or analysis tools, and performed the analysis; P.D.-R. and E.G.M. collected the data and performed the analysis; T.N. performed the analysis; F.A.G.P., T.I.R., E.M.M., T.S., K.L., R.M., T.G., Y.W., J.P.A., R.B., N.M.A.-S., G.R., I.E.T., K.P., F.L., P.M., K.S., K.E.G.M., L.A., A.R.C., A.A., T.B.-H., T.R.R., F.I., P.F.J., K.H.M.-H. and Y.Z. contributed data or analysis tools; C.E. and A.U. collected the data, other contribution (annotated data from ground truth); E.M. and A.C. conceived and designed the analysis; and A.M.-B., M.K. and C.O.S. conceived and designed the analysis and wrote the paper. Data availability. The training datasets with their reference annotations and test datasets used in the challenge are publicly available at the CTC website (http://celltrackingchallenge.net). With regard to the raw data for individual Figures and Tables, Source data are provided with this paper. Code availability. The evaluation routines used to produce the results reported in this article are freely available at the CTC website or the CellTrackingChallenge update site as a Fiji plugin, the source codes of which can be found at https://github.com/CellTrackingChallenge/. Furthermore, this public GitHub repository contains links to the executable versions of the individual algorithms and Colab Notebooks of those 11 participants who agreed to share their tools in a reusable form. The parameters used by the participants to produce their benchmarked results are listed on the CTC website. Competing interests. K.S. is employed part-time by LPIXEL Inc. All other authors have no competing interests.

Attached Files

In Press - s41592-023-01879-y.pdf


Files (9.6 MB)
Name Size Download all
9.6 MB Preview Download

Additional details

August 22, 2023
October 20, 2023