CaltechAUTHORS
  A Caltech Library Service

The Zwicky Transient Facility Alert Distribution System

Patterson, Maria T. and Bellm, Eric C. and Rusholme, Ben and Masci, Frank J. and Juric, Mario and Krughoff, K. Simon and Golkhou, V. Zach and Graham, Matthew J. and Kulkarni, Shrinivas R. and Helou, George (2019) The Zwicky Transient Facility Alert Distribution System. Publications of the Astronomical Society of the Pacific, 131 (995). Art. No. 018001. ISSN 0004-6280. https://resolver.caltech.edu/CaltechAUTHORS:20181127-101530598

[img] PDF - Submitted Version
See Usage Policy.

777Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20181127-101530598

Abstract

The Zwicky Transient Facility (ZTF) survey generates real-time alerts for optical transients, variables, and moving objects discovered in its wide-field survey. We describe the ZTF alert stream distribution and processing (filtering) system. The system uses existing open-source technologies developed in industry: Kafka, a real-time streaming platform, and Avro, a binary serialization format. The technologies used in this system provide a number of advantages for the ZTF use case, including (1) built-in replication, scalability, and stream rewind for the distribution mechanism; (2) structured messages with strictly enforced schemas and dynamic typing for fast parsing; and (3) a Python-based stream processing interface that is similar to batch for a familiar and user-friendly plug-in filter system, all in a modular, primarily containerized system. The production deployment has successfully supported streaming up to 1.2 million alerts or roughly 70 GB of data per night, with each alert available to a consumer within about 10 s of alert candidate production. Data transfer rates of about 80,000 alerts/minute have been observed. In this paper, we discuss this alert distribution and processing system, the design motivations for the technology choices for the framework, performance in production, and how this system may be generally suitable for other alert stream use cases, including the upcoming Large Synoptic Survey Telescope.


Item Type:Article
Related URLs:
URLURL TypeDescription
https://doi.org/10.1088/1538-3873/aae904DOIArticle
https://arxiv.org/abs/1902.02227arXivDiscussion Paper
ORCID:
AuthorORCID
Patterson, Maria T.0000-0002-4753-3387
Bellm, Eric C.0000-0001-8018-5348
Rusholme, Ben0000-0001-7648-4142
Masci, Frank J.0000-0002-8532-9395
Juric, Mario0000-0003-1996-9252
Krughoff, K. Simon0000-0002-4410-7868
Golkhou, V. Zach0000-0001-8205-2506
Graham, Matthew J.0000-0002-3168-0139
Kulkarni, Shrinivas R.0000-0001-5390-8563
Helou, George0000-0003-3367-3415
Additional Information:© 2018 The Astronomical Society of the Pacific. Received 2018 September 14; accepted 2018 October 10; published 2018 November 27. Based on observations obtained with the Samuel Oschin Telescope 48 inch and the 60 inch Telescope at the Palomar Observatory as part of the Zwicky Transient Facility project, a scientific collaboration among the California Institute of Technology, the Oskar Klein Centre, the Weizmann Institute of Science, the University of Maryland, the University of Washington, Deutsches Elektronen-Synchrotron, the University of Wisconsin-Milwaukee, and the TANGO Program of the University System of Taiwan. Further support is provided by the U.S. National Science Foundation under grant No. AST-1440341. M. Patterson, E. Bellm., and K. S. Krughoff acknowledge support from the Large Synoptic Survey Telescope program, which is supported in part by the National Science Foundation through Cooperative Agreement 1258333 managed by the Association of Universities for Research in Astronomy (AURA), and the Department of Energy under contract No. DE-AC02-76SF00515 with the SLAC National Accelerator Laboratory. Additional LSST funding comes from private donations, grants to universities, and in-kind support from LSSTC Institutional Members. M. Patterson, E. Bellm, V. Z. Golkhou, and M. Juric acknowledge support from the University of Washington College of Arts and Sciences, Department of Astronomy, and the DIRAC Institute. University of Washington's DIRAC Institute is supported through generous gifts from the Charles and Lisa Simonyi Fund for Arts and Sciences, and the Washington Research Foundation. E. Bellm is supported in part by the NSF AAG grant 1812779 and grant #2018-0908 from the Heising-Simons Foundation. M. Juric acknowledges the support of the Washington Research Foundation Data Science Term Chair fund, and the UW Provost's Initiative in Data-Intensive Discovery.
Group:Infrared Processing and Analysis Center (IPAC), Zwicky Transient Facility
Funders:
Funding AgencyGrant Number
NSFAST-1440341
NSFAST-1258333
Department of Energy (DOE)DE-AC02-76SF00515
University of WashingtonUNSPECIFIED
DIRAC InstituteUNSPECIFIED
NSFAST-1812779
Heising-Simons Foundation2018-0908
Washington Research FoundationUNSPECIFIED
Subject Keywords:astronomical databases: miscellaneous – instrumentation: iscellaneous – surveys
Issue or Number:995
Record Number:CaltechAUTHORS:20181127-101530598
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20181127-101530598
Official Citation:Maria T. Patterson et al 2019 PASP 131 018001
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:91216
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:27 Nov 2018 18:25
Last Modified:03 Oct 2019 20:32

Repository Staff Only: item control page