CaltechAUTHORS
  A Caltech Library Service

Unicorn: Unified resource orchestration for multi-domain, geo-distributed data analytics

Xiang, Qiao and Chen, Shenshen and Gao, Kai and Newman, Harvey and Taylor, Ian and Zhang, Jingxuan and Yang, Yang Richard (2017) Unicorn: Unified resource orchestration for multi-domain, geo-distributed data analytics. In: 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). IEEE , Piscataway, NJ, pp. 1-6. ISBN 978-1-5386-0435-9. https://resolver.caltech.edu/CaltechAUTHORS:20180706-110608219

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20180706-110608219

Abstract

Data-intensive analytics is entering the era of multi-organizational, geographically-distributed, collaborative computing, where different organizations contribute various resources, e.g., sensing, computation, storage and networking resources, to collaboratively collect, share and analyze extremely large amounts of data. This new paradigm calls for a framework to manage a large set of distributively owned heterogeneous resources, with the fundamental objective of efficient resource utilization, following the autonomy and privacy of resource owners. In this paper, we design Unicorn, the first unified framework that accomplishes this goal. The foundation of Unicorn is RSDP, an autonomous, privacy-preserving resource discovery and representation system to provide accurate resource availability information. Its core is a novel abstraction called resource vector abstraction which describes the resource availability in a set of linear constraints. In addition, Unicorn also provides a series of advanced solutions to support automatic, efficient management of resource dynamics on both supply and demand sides, including an automatic workflow transformer, an intelligent resource demand estimator and an efficient, scalable multi-resource orchestrator. Being the first unified framework for this new paradigm, Unicorn plays a fundamental role in next-generation data-intensive collaborative computing systems.


Item Type:Book Section
Related URLs:
URLURL TypeDescription
https://doi.org/10.1109/UIC-ATC.2017.8397409DOIArticle
ORCID:
AuthorORCID
Newman, Harvey0000-0003-0964-1480
Additional Information:© 2018 IEEE. We thank Justas Balcas, Shiwei Chen, Lili Liu, Maria Spiropulu and Jean-Roch Vlimant for helpful discussion during the work. The Yale team was supported in part by NSF grant #1440745, CC*IIE Integration: Dynamically Optimizing Research Data Workflow with a Software Defined Science Network; International Technology Alliance Agreement No W911NF-16-3-0002; Google Research Award, SDN Programming Using just Minimal Abstractions; NSFC #61672385, FAST Magellan. The Tongji team was supported by China Postdoctoral Science Foundation #2017M611618. The Caltech team was supported in part by DOE/ASCR project #000219898, SDN NGenIA; DOE award #DE-AC02-07CHI1359, SENSE, FNAL PO #626507; NSF award #1246133, ANES; NSF award #1341024, CHOPIN.
Funders:
Funding AgencyGrant Number
NSFOAC-1440745
Army Research LaboratoryW911NF-16-3-0002
GoogleUNSPECIFIED
National Natural Science Foundation of China61672385
Chinese Postdoctoral Science Foundation2017M611618
Department of Energy (DOE)000219898
Department of Energy (DOE)DE-AC02-07CHI1359
NSFOAC-1246133
NSFOAC-1341024
Subject Keywords:multi-domain, data analytics, resource allocation
Record Number:CaltechAUTHORS:20180706-110608219
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20180706-110608219
Official Citation:Q. Xiang et al., "Unicorn: Unified resource orchestration for multi-domain, geo-distributed data analytics," 2017 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computed, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI), San Francisco, CA, USA, 2017, pp. 1-6. doi: 10.1109/UIC-ATC.2017.8397409
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:87597
Collection:CaltechAUTHORS
Deposited By: George Porter
Deposited On:06 Jul 2018 22:02
Last Modified:03 Oct 2019 19:57

Repository Staff Only: item control page