CaltechAUTHORS
  A Caltech Library Service

Scheduling Data-Intensive Workflows onto Storage-Constrained Distributed Resources

Ramakrishnan, Arun and Singh, Gurmeet and Zhao, Henan and Deelman, Ewa and Sakellariou, Rizos and Vahi, Karan and Blackburn, Kent and Meyers, David and Samidi, Michael (2007) Scheduling Data-Intensive Workflows onto Storage-Constrained Distributed Resources. In: Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid '07). IEEE , Piscataway, NJ, pp. 401-409. ISBN 0-7695-2833-3. http://resolver.caltech.edu/CaltechAUTHORS:20170419-154436528

[img] PDF - Published Version
See Usage Policy.

535Kb

Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20170419-154436528

Abstract

In this paper we examine the issue of optimizing disk usage and of scheduling large-scale scientific workflows onto distributed resources where the workflows are data- intensive, requiring large amounts of data storage, and where the resources have limited storage resources. Our approach is two-fold: we minimize the amount of space a workflow requires during execution by removing data files at runtime when they are no longer required and we schedule the workflows in a way that assures that the amount of data required and generated by the workflow fits onto the individual resources. For a workflow used by gravitational- wave physicists, we were able to improve the amount of storage required by the workflow by up to 57 %. We also designed an algorithm that can not only find feasible solutions for workflow task assignment to resources in disk- space constrained environments, but can also improve the overall workflow performance.


Item Type:Book Section
Related URLs:
URLURL TypeDescription
http://dx.doi.org/0.1109/CCGRID.2007.101DOIArticle
http://ieeexplore.ieee.org/document/4215405/PublisherArticle
Additional Information:© 2007 IEEE. This work was supported by the National Science Foundation under the grant CNS 0615412. R. Sakellariou and H. Zhao would like to acknowledge partial support from the EU-funded CoreGrid Network of Excellence (grant FP6-004265) and the UK EPSRC grant GR/S67654/01. The authors thank Duncan Brown for his contributions to the LIGO workflow used to model simulated workflows. The authors also thank the Open Science Grid for resources used to motivate the work presented. The work of K. Blackburn, D. Meyers. and M. Samidi was supported by the National Science Foundation under awards PHY-0107417 and PHY-0326281. The LIGO Observatories were constructed by the California Institute of Technology and Massachusetts Institute of Technology with funding from the National Science Foundation under cooperative agreement PHY-9210038. The LIGO Laboratory operates under cooperative agreement PHY-0107417. This paper has been assigned LIGO Document Number LIGO-P070003-00-Z.
Group:LIGO
Funders:
Funding AgencyGrant Number
NSFCNS 0615412
CoreGrid Network of ExcellenceFP6-004265
Engineering and Physical Sciences Research Council (EPSRC)GR/S67654/01
NSFPHY-0107417
NSFPHY-0326281
NSFPHY-9210038
NSFPHY-0107417
Other Numbering System:
Other Numbering System NameOther Numbering System ID
LIGO DocumentLIGO-P070003-00-Z
Record Number:CaltechAUTHORS:20170419-154436528
Persistent URL:http://resolver.caltech.edu/CaltechAUTHORS:20170419-154436528
Official Citation:A. Ramakrishnan et al., "Scheduling Data-IntensiveWorkflows onto Storage-Constrained Distributed Resources," Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid '07), Rio De Janeiro, 2007, pp. 401-409. doi: 10.1109/CCGRID.2007.101
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:76722
Collection:CaltechAUTHORS
Deposited By: Kristin Buxton
Deposited On:19 Apr 2017 23:03
Last Modified:19 Apr 2017 23:03

Repository Staff Only: item control page