A Caltech Library Service

Photometric LSST Astronomical Time-series Classification Challenge PLAsTiCC: Selection of a Performance Metric for Classification Probabilities Balancing Diverse Science Goals

Malz, A. I. and Hložek, R. and Allam, T., Jr. and Bahmanyar, A. and Biswas, R. and Dai, M. and Galbany, L. and Ishida, E. E. O. and Jha, S. W. and Jones, D. O. and Kessler, R. and Lochner, M. and Mahabal, A. A. and Mandel, K. S. and Martínez-Galarza, J. R. and McEwen, J. D. and Muthukrishna, D. and Narayan, G. and Peiris, H. and Peters, C. M. and Ponder, K. and Setzer, C. N. (2019) Photometric LSST Astronomical Time-series Classification Challenge PLAsTiCC: Selection of a Performance Metric for Classification Probabilities Balancing Diverse Science Goals. Astronomical Journal, 158 (5). Art. No. 171. ISSN 1538-3881.

[img] PDF - Published Version
See Usage Policy.

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


Classification of transient and variable light curves is an essential step in using astronomical observations to develop an understanding of the underlying physical processes from which they arise. However, upcoming deep photometric surveys, including the Large Synoptic Survey Telescope (LSST), will produce a deluge of low signal-to-noise data for which traditional type estimation procedures are inappropriate. Probabilistic classification is more appropriate for such data but is incompatible with the traditional metrics used on deterministic classifications. Furthermore, large survey collaborations like LSST intend to use the resulting classification probabilities for diverse science objectives, indicating a need for a metric that balances a variety of goals. We describe the process used to develop an optimal performance metric for an open classification challenge that seeks to identify probabilistic classifiers that can serve many scientific interests. The Photometric LSST Astronomical Time-series Classification Challenge (PLAsTiCC) aims to identify promising techniques for obtaining classification probabilities of transient and variable objects by engaging a broader community beyond astronomy. Using mock classification probability submissions emulating realistically complex archetypes of those anticipated of PLAsTiCC, we compare the sensitivity of two metrics of classification probabilities under various weighting schemes, finding that both yield results that are qualitatively consistent with intuitive notions of classification performance. We thus choose as a metric for PLAsTiCC a weighted modification of the cross-entropy because it can be meaningfully interpreted in terms of information content. Finally, we propose extensions of our methodology to ever more complex challenge goals and suggest some guiding principles for approaching the choice of a metric of probabilistic data products.

Item Type:Article
Related URLs:
URLURL TypeDescription Paper
Malz, A. I.0000-0002-8676-1622
Hložek, R.0000-0002-0965-7864
Allam, T., Jr.0000-0002-8117-9254
Biswas, R.0000-0002-5741-7195
Galbany, L.0000-0002-1296-6887
Ishida, E. E. O.0000-0002-0406-076X
Jha, S. W.0000-0001-8738-6011
Jones, D. O.0000-0002-6230-0151
Kessler, R.0000-0003-3221-0419
Lochner, M.0000-0003-2221-8281
Mahabal, A. A.0000-0003-2242-0244
Mandel, K. S.0000-0001-9846-4417
Martínez-Galarza, J. R.0000-0002-5069-0324
McEwen, J. D.0000-0002-5852-8890
Muthukrishna, D.0000-0002-5788-9280
Narayan, G.0000-0001-6022-0484
Peiris, H.0000-0002-2519-584X
Peters, C. M.0000-0003-4639-0211
Ponder, K.0000-0002-8207-3304
Setzer, C. N.0000-0002-7439-2735
Additional Information:© 2019 The American Astronomical Society. Received 2018 October 18; revised 2019 July 31; accepted 2019 August 8; published 2019 October 10. Author contributions are listed below. A.I.M.: conceptualization, data curation, formal analysis, investigation, methodology, project administration, software, supervision, validation, visualization, writing—editing, writing—original draft R.H.: data curation, formal analysis, funding acquisition, investigation, project administration, software, supervision, validation, visualization, writing—editing, writing—original draft T.A.Jr.: investigation, software, validation, writing—original draft A.B.: formal analysis, investigation, methodology, software, writing—editing, writing—original draft R.B.: conceptualization, methodology, software, supervision, writing—editing, writing—original draft M.D.: writing—editing L.G.: writing—editing E.E.O.I.: conceptualization, project administration, supervision, writing—editing S.W.J.: writing—editing D.J.: software R.K.: writing—editing M.L.: conceptualization, data curation, formal analysis, visualization, writing—editing A.A.M.: data curation, software, writing—editing, writing—original draft K.S.M.: conceptualization, supervision, writing—editing J.R.M-G.: data curation, software, visualization, writing—original draft J.D.M.: conceptualization, investigation, supervision D.M.: data curation, validation G.N.: data curation, formal analysis H.P.: conceptualization, funding acquisition, supervision C.M.P.: writing—editing K.P.: visualization, writing—editing C.N.S.: conceptualization, software. This paper has undergone internal review in the LSST Dark Energy Science Collaboration. The authors would like to thank Melissa Graham, Weikang Lin, and Chad Schafer for serving as the LSST-DESC publication review committee, as well as Tom Loredo for other helpful feedback. The authors also express gratitude to the anonymous referee for substantive suggestions that improved the paper. A.I.M. was advised by David W. Hogg and was supported by National Science Foundation grant AST-1517237. A.I.M. acknowledges support from the Max Planck Society and the Alexander von Humboldt Foundation in the framework of the Max Planck-Humboldt Research Award endowed by the Federal Ministry of Education and Research. T.A. is supported in part by STFC. R.B. and C.S. are supported by the Swedish Research Council (VR) through the Oskar Klein Centre. Their work was further supported by the research environment grant "Gravitational Radiation and Electromagnetic Astrophysical Transients (GREAT)" funded by the Swedish Research council (VR) under Dnr 2016-06012. A.A.M. was supported in part by the NSF grants AST-0909182, AST-1313422, AST-1413600, and AST-1518308, and by the Ajax Foundation. E. E. O. I. acknowledges support from CNRS 2017 MOMENTUM grant. D.O.J. is supported by a Gordon and Betty Moore Foundation postdoctoral fellowship at the University of California, Santa Cruz. L.G. was funded by the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 839090. The financial assistance of the National Research Foundation (NRF) toward this research is hereby acknowledged. Opinions expressed and conclusions arrived at, are those of the authors and are not necessarily to be attributed to the NRF. This work is partially supported by the European Research Council under the European Communitys Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no 306478-CosmicDawn. Canadian co-authors acknowledge support from the Natural Sciences and Engineering Research Council of Canada. The Dunlap Institute is funded through an endowment established by the David Dunlap family and the University of Toronto. The authors at the University of Toronto acknowledge that the land on which the University of Toronto is built is the traditional territory of the Haudenosaunee, and most recently, the territory of the Mississaugas of the New Credit First Nation. They are grateful to have the opportunity to work in the community, on this territory. We acknowledge the University of Chicago Research Computing Center for support of this work. This research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility operated under Contract No. DE-AC02-05CH11231. This research at Rutgers University is supported by US Department of Energy award DE-SC0011636. The DESC acknowledges ongoing support from the Institut National de Physique Nucléaire et de Physique des Particules in France; the Science & Technology Facilities Council in the United Kingdom; and the Department of Energy, the National Science Foundation, and the LSST Corporation in the United States. DESC uses resources of the IN2P3 Computing Center (CC-IN2P3–Lyon/Villeurbanne—France) funded by the Centre National de la Recherche Scientifique; the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231; STFC DiRAC HPC Facilities, funded by UK BIS National E-infrastructure capital grants; and the UK particle physics grid, supported by the GridPP Collaboration. This work was performed in part under DOE Contract DE-AC02-76SF00515. Software: jupyter (Kluyver et al. 2016), matplotlib (Hunter 2007), numpy (Oliphant 2006, 2007; van der Walt et al. 2011), proclam (Malz 2018), scikit-learn (Pedregosa et al. 2011), scipy (Jones et al. 2001; Buitinck et al. 2013).
Funding AgencyGrant Number
Max Planck SocietyUNSPECIFIED
Alexander von Humboldt FoundationUNSPECIFIED
Bundesministerium für Bildung und Forschung (BMBF)UNSPECIFIED
Science and Technology Facilities Council (STFC)UNSPECIFIED
Swedish Research CouncilDnr 2016-06012
Ajax FoundationUNSPECIFIED
Centre National de la Recherche Scientifique (CNRS)UNSPECIFIED
Gordon and Betty Moore FoundationUNSPECIFIED
Marie Curie Fellowship839090
National Research Foundation of KoreaUNSPECIFIED
European Research Council (ERC)306478
Natural Sciences and Engineering Research Council of Canada (NSERC)UNSPECIFIED
Dunlap InstituteUNSPECIFIED
University of TorontoUNSPECIFIED
Department of Energy (DOE)DE-AC02-05CH11231
Department of Energy (DOE)DE-SC0011636
Department of Energy (DOE)DE-AC02-76SF00515
Subject Keywords:methods: data analysis – methods: statistical – stars: variables: general – supernovae: general – surveys – techniques: photometric
Issue or Number:5
Record Number:CaltechAUTHORS:20191010-133618114
Persistent URL:
Official Citation:A. I. Malz et al 2019 AJ 158 171
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:99217
Deposited By: Tony Diaz
Deposited On:10 Oct 2019 20:46
Last Modified:10 Oct 2019 20:46

Repository Staff Only: item control page