CaltechAUTHORS
  A Caltech Library Service

Astronomical Image Processing at Scale With Pegasus and Montage

Berriman, G. Bruce and Good, John C. and Deelman, Ewa and Tanaka, Ryan and Vahi, Karan (2021) Astronomical Image Processing at Scale With Pegasus and Montage. . (Unpublished) https://resolver.caltech.edu/CaltechAUTHORS:20220329-173612262

[img] PDF - Accepted Version
Creative Commons Attribution.

138kB

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20220329-173612262

Abstract

Image processing at scale is a powerful tool for creating new data sets and integrating them with existing data sets and performing analysis and quality assurance investigations. Workflow managers offer advantages in this type of processing, which involves multiple data access and processing steps. Generally, they enable automation of the workflow by locating data and resources, recovery from failures, and monitoring of performance. In this focus demo we demonstrate how the Pegasus Workflow Manager Python API manages image processing to create mosaics with the Montage Image Mosaic engine. Since 2001, Pegasus has been developed and maintained at USC/ISI. Montage was in fact one of the first applications used to design Pegasus and optimize its performance. Pegasus has since found application in many areas of science. LIGO exploited it in making discoveries of black holes. The Vera C. Rubin Observatory used it to compare the cost and performance of processing images on cloud platforms. While these are examples of projects at large scale, small team investigations on local clusters of machines can benefit from Pegasus as well.


Item Type:Report or Paper (Discussion Paper)
Related URLs:
URLURL TypeDescription
http://arxiv.org/abs/2111.11624arXivDiscussion Paper
ORCID:
AuthorORCID
Berriman, G. Bruce0000-0001-8388-534X
Deelman, Ewa0000-0001-5106-503X
Vahi, Karan0000-0001-8622-2082
Additional Information:Attribution 4.0 International (CC BY 4.0) Pegasus is funded by the National Science Foundation under OAC SI2-SSI program grant 1664162. Previously, NSF funded Pegasus under OCI SDCI program grant 0722019 and OCI SI2-SSI program grant 1148515. Montage is funded by the National Science Foundation grant awards 1835379, 1642453 and 1440620. OSG is supported by the National Science Foundation grant award 2030508.
Group:Infrared Processing and Analysis Center (IPAC)
Funders:
Funding AgencyGrant Number
NSFOAC-1664162
NSFOCI-0722019
NSFOCI-1148515
NSFOAC-1835379
NSFOAC-1642453
NSFOAC-1440620
NSFOAC-2030508
Record Number:CaltechAUTHORS:20220329-173612262
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20220329-173612262
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:114110
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:30 Mar 2022 19:12
Last Modified:30 Mar 2022 19:12

Repository Staff Only: item control page