CaltechAUTHORS
  A Caltech Library Service

The NOAO Data Lab virtual storage system

Graham, Matthew J. and Fitzpatrick, Michael J. and Norris, Patrick and Mighell, Kenneth J. and Olsen, Knut and Stobie, Elizabeth B. and Ridgway, Stephen T. and Bolton, Adam S. and Saha, Abhijit and Huang, Lijuan (2016) The NOAO Data Lab virtual storage system. In: Software and Cyberinfrastructure for Astronomy IV. Proceedings of SPIE. No.9913. Society of Photo-Optical Instrumentation Engineers (SPIE) , Bellingham, WA, Art. No. 99131I. ISBN 978-1-5106-0205-2. https://resolver.caltech.edu/CaltechAUTHORS:20170110-115645930

[img] PDF - Published Version
See Usage Policy.

417Kb

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20170110-115645930

Abstract

Collaborative research/computing environments are essential for working with the next generations of large astronomical data sets. A key component of them is a distributed storage system to enable data hosting, sharing, and publication. VOSpace is a lightweight interface providing network access to arbitrary backend storage solutions and endorsed by the International Virtual Observatory Alliance (IVOA). Although similar APIs exist, such as Amazon S3, WebDav, and Dropbox, VOSpace is designed to be protocol agnostic, focusing on data control operations, and supports asynchronous and third-party data transfers, thereby minimizing unnecessary data transfers. It also allows arbitrary computations to be triggered as a result of a transfer operation: for example, a file can be automatically ingested into a database when put into an active directory or a data reduction task, such as Sextractor, can be run on it. In this paper, we shall describe the VOSpace implementations that we have developed for the NOAO Data Lab. These offer both dedicated remote storage, accessible as a local file system via FUSE, and a local VOSpace service to easily enable data synchronization.


Item Type:Book Section
Related URLs:
URLURL TypeDescription
http://dx.doi.org/10.1117/12.2233306DOIArticle
ORCID:
AuthorORCID
Graham, Matthew J.0000-0002-3168-0139
Olsen, Knut0000-0002-7134-8296
Saha, Abhijit0000-0002-6839-4881
Additional Information:© 2016 Society of Photo-Optical Instrumentation Engineers (SPIE). NOAO is operated by the Association of Universities for Research in Astronomy (AURA). under cooperative agreement with the National Science Foundation.
Funders:
Funding AgencyGrant Number
Association of Universities for Research in Astronomy (AURA)UNSPECIFIED
NSFUNSPECIFIED
Subject Keywords:Science exploration framework, Virtual storage, data services, remote execution
Series Name:Proceedings of SPIE
Issue or Number:9913
Record Number:CaltechAUTHORS:20170110-115645930
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20170110-115645930
Official Citation:Matthew J. Graham ; Michael J. Fitzpatrick ; Patrick Norris ; Kenneth J. Mighell ; Knut Olsen ; Elizabeth B. Stobie ; Stephen T. Ridgway ; Adam S. Bolton ; Abhijit Saha ; Lijuan W. Huang; The NOAO Data Lab virtual storage system. Proc. SPIE 9913, Software and Cyberinfrastructure for Astronomy IV, 99131I (July 26, 2016); doi:10.1117/12.2233306.
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:73377
Collection:CaltechAUTHORS
Deposited By: Ruth Sustaita
Deposited On:10 Jan 2017 21:56
Last Modified:17 Nov 2019 06:10

Repository Staff Only: item control page