A Caltech Library Service

Streamlining Pipeline Workflows: Using Python with an Object-Oriented Approach to Consolidate Aggregate Pipeline Processes

Brown, M. K. and Mader, J. A. and Berriman, G. B. and Gelino, C. R. and Kong, M. and Laity, A. C. and Riley, J. and Rizzi, L. and Swain, M. A. (2019) Streamlining Pipeline Workflows: Using Python with an Object-Oriented Approach to Consolidate Aggregate Pipeline Processes. In: Astronomical Data Analysis Software and Systems XXVIII. Astronomical Society of the Pacific Conference Series. No.523. Astronomical Society of the Pacific , San Francisco, CA, pp. 163-166. ISBN 978-1-58381-933-3.

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item:


The Keck Observatory Archive (KOA), a collaboration between the NASA Exoplanet Science Institute and the W. M. Keck Observatory, serves science and calibration data for all current and retired instruments from the twin Keck Telescopes. In addition to the raw data, we publicly serve quick-look, reduced data products for four instruments (HIRES, LWS, NIRC2, NIRSPEC and OSIRIS), so that KOA users can easily assess the quality and scientific content of the data. In this paper we present the modernization of the Data Evaluation and Processing (DEP) Pipeline, our quality assurance tool to ensure science data is ready for archiving. Since there was no common infrastructure for data headers, the DEP pipeline had to evolve to accommodate new instruments through additional control paths each time an instrument was added or upgraded. Over time, new modules to assist with the processing were added in a variety of languages including IDL, C, CSH, PHP, and Python. The calls to multiple interpreters caused a lot of overhead. This project was an initiative to consolidate the DEP pipeline into a common language, Python, using an object-oriented approach. The object-oriented approach allows us to abstract out the differences and use common variables in place of instrument-specific values. As a result, new instruments only need a modified subclass with the differing values in order to work with the pipeline. By consolidating everything to Python, we have seen an increase in efficiency, ease of operation, and ease of maintenance.

Item Type:Book Section
Related URLs:
URLURL TypeDescription
Berriman, G. B.0000-0001-8388-534X
Rizzi, L.0000-0003-0882-2327
Additional Information:© 2019 Astronomical Society of the Pacific. KOA is a collaboration between the W. M. Keck Observatory (WMKO) and the NASA Exoplanet Science Institute (NExScI). Funding for KOA is provided by NASA under award No. 80NSSC18M0066. WMKO is operated as a scientific partnership among the California Institute of Technology, the University of California and NASA.
Group:Infrared Processing and Analysis Center (IPAC)
Funding AgencyGrant Number
Series Name:Astronomical Society of the Pacific Conference Series
Issue or Number:523
Record Number:CaltechAUTHORS:20191202-085935454
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:100127
Deposited By: Tony Diaz
Deposited On:02 Dec 2019 17:07
Last Modified:02 Dec 2019 17:07

Repository Staff Only: item control page