A Caltech Library Service

Data and Metadata Management at the Keck Observatory Archive

Berriman, G. B. and Holt, J. M. and Mader, J. A. and Tran, H. D. and Goodrich, R. W. and Gelino, C. R. and Laity, A. C. and Kong, M. and Swain, M. A. (2015) Data and Metadata Management at the Keck Observatory Archive. In: Astronomical Data Analysis Software and Systems XXIV. Astronomical Society of the Pacific conference series. No.495. Astronomical Society of the Pacific , San Francisco, CA, pp. 535-538. ISBN 9781583818749.

[img] PDF - Published Version
See Usage Policy.

[img] PDF - Submitted Version
See Usage Policy.


Use this Persistent URL to link to this item:


A collaboration between the W. M. Keck Observatory (WMKO) in Hawaii and the NASA Exoplanet Science Institute (NExScI) in California, the Keck Observatory Archive (KOA) was commissioned in 2004 to archive data from WMKO, which operates two classically scheduled 10 m ground-based telescopes. The data from Keck are not suitable for direct ingestion into the archive since the metadata contained in the original FITS headers lack the information necessary for proper archiving. The data pose a number of challenges for KOA: different instrument builders used different standards, and the nature of classical observing, where observers have complete control of the instruments and their observations, lead to heterogeneous data sets. For example, it is often difficult to determine if an observation is a science target, a sky frame, or a sky flat. It is also necessary to assign the data to the correct owners and observing programs, which can be a challenge for time-domain and target-of-opportunity observations, or on split nights, during which two or more principle investigators share a given night. In addition, having uniform and adequate calibrations is important for the proper reduction of data. Therefore, KOA needs to distinguish science files from calibration files, identify the type of calibrations available, and associate the appropriate calibration files with each science frame. We describe the methodologies and tools that we have developed to successfully address these difficulties, adding content to the FITS headers and “retrofitting" the metadata in order to support archiving Keck data, especially those obtained before the archive was designed. With the expertise gained from having successfully archived observations taken with all eight currently active instruments at WMKO, we have developed lessons learned from handling this complex array of heterogeneous metadata. These lessons help ensure a smooth ingestion of data not only for current but also future instruments, as well as a better experience for the archive user.

Item Type:Book Section
Related URLs:
URLURL TypeDescription Paper
Berriman, G. B.0000-0001-8388-534X
Alternate Title:Metadata and Data Management for the Keck Observatory Archive
Additional Information:© 2015 Astronomical Society of the Pacific. The Keck Observatory Archive (KOA) is collaboration between the W. M. Keck Observatory (WMKO) and the NASA Exoplanet Science Institute (NExScI). Funding for KOA is provided by the National Aeronautics and Space Administration (NASA). WMKO is operated as a scientific partnership among the California Institute of Technology, the University of California, and NASA. The Observatory was made possible by the generous financial support of the W. M. Keck Foundation. NExScI is sponsored by NASA's Origins Theme and Exoplanet Exploration Program, and operated by the California Institute of Technology in coordination with the Jet Propulsion Laboratory. This material is based in part upon work supported by NASA under Grant and Cooperative agreement No. NNX13AH26A.
Group:Infrared Processing and Analysis Center (IPAC)
Funding AgencyGrant Number
W. M. Keck FoundationUNSPECIFIED
Series Name:Astronomical Society of the Pacific conference series
Issue or Number:495
Record Number:CaltechAUTHORS:20160325-081133335
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:65667
Deposited By: Tony Diaz
Deposited On:30 Mar 2016 23:35
Last Modified:28 Oct 2019 20:53

Repository Staff Only: item control page