Brown, M. K. and Mader, J. A. and Berriman, G. B. and Gelino, C. R. and Kong, M. and Laity, A. C. and Riley, J. and Rizzi, L. and Swain, M. A. (2019) Streamlining Pipeline Workflows: Using Python with an Object-Oriented Approach to Consolidate Aggregate Pipeline Processes. In: Astronomical Data Analysis Software and Systems XXVIII. Astronomical Society of the Pacific Conference Series. No.523. Astronomical Society of the Pacific , San Francisco, CA, pp. 163-166. ISBN 978-1-58381-933-3. https://resolver.caltech.edu/CaltechAUTHORS:20191202-085935454
Full text is not posted in this repository. Consult Related URLs below.
Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20191202-085935454
Abstract
The Keck Observatory Archive (KOA), a collaboration between the NASA Exoplanet Science Institute and the W. M. Keck Observatory, serves science and calibration data for all current and retired instruments from the twin Keck Telescopes. In addition to the raw data, we publicly serve quick-look, reduced data products for four instruments (HIRES, LWS, NIRC2, NIRSPEC and OSIRIS), so that KOA users can easily assess the quality and scientific content of the data. In this paper we present the modernization of the Data Evaluation and Processing (DEP) Pipeline, our quality assurance tool to ensure science data is ready for archiving. Since there was no common infrastructure for data headers, the DEP pipeline had to evolve to accommodate new instruments through additional control paths each time an instrument was added or upgraded. Over time, new modules to assist with the processing were added in a variety of languages including IDL, C, CSH, PHP, and Python. The calls to multiple interpreters caused a lot of overhead. This project was an initiative to consolidate the DEP pipeline into a common language, Python, using an object-oriented approach. The object-oriented approach allows us to abstract out the differences and use common variables in place of instrument-specific values. As a result, new instruments only need a modified subclass with the differing values in order to work with the pipeline. By consolidating everything to Python, we have seen an increase in efficiency, ease of operation, and ease of maintenance.
| Item Type: | Book Section | ||||||
|---|---|---|---|---|---|---|---|
| Related URLs: |
| ||||||
| ORCID: |
| ||||||
| Additional Information: | © 2019 Astronomical Society of the Pacific. KOA is a collaboration between the W. M. Keck Observatory (WMKO) and the NASA Exoplanet Science Institute (NExScI). Funding for KOA is provided by NASA under award No. 80NSSC18M0066. WMKO is operated as a scientific partnership among the California Institute of Technology, the University of California and NASA. | ||||||
| Group: | Infrared Processing and Analysis Center (IPAC) | ||||||
| Funders: |
| ||||||
| Series Name: | Astronomical Society of the Pacific Conference Series | ||||||
| Issue or Number: | 523 | ||||||
| Record Number: | CaltechAUTHORS:20191202-085935454 | ||||||
| Persistent URL: | https://resolver.caltech.edu/CaltechAUTHORS:20191202-085935454 | ||||||
| Usage Policy: | No commercial reproduction, distribution, display or performance rights in this work are provided. | ||||||
| ID Code: | 100127 | ||||||
| Collection: | CaltechAUTHORS | ||||||
| Deposited By: | Tony Diaz | ||||||
| Deposited On: | 02 Dec 2019 17:07 | ||||||
| Last Modified: | 02 Dec 2019 17:07 |
Repository Staff Only: item control page
