CaltechAUTHORS
  A Caltech Library Service

High Throughput WAN Data Transfer with Hadoop-based Storage

Amin, A. and Bockelman, B. and Letts, J. and Levshina, T. and Martin, T. and Pi, H. and Sfiligoi, I. and Thomas, M. and Wüerthwein, F. (2011) High Throughput WAN Data Transfer with Hadoop-based Storage. Journal of Physics Conference Series, 331 . Art. No. 052016. ISSN 1742-6588 . https://resolver.caltech.edu/CaltechAUTHORS:20120410-110725216

Full text is not posted in this repository. Consult Related URLs below.

Use this Persistent URL to link to this item: https://resolver.caltech.edu/CaltechAUTHORS:20120410-110725216

Abstract

Hadoop distributed file system (HDFS) is becoming more popular in recent years as a key building block of integrated grid storage solution in the field of scientific computing. Wide Area Network (WAN) data transfer is one of the important data operations for large high energy physics experiments to manage, share and process datasets of PetaBytes scale in a highly distributed grid computing environment. In this paper, we present the experience of high throughput WAN data transfer with HDFS-based Storage Element. Two protocols, GridFTP and fast data transfer (FDT), are used to characterize the network performance of WAN data transfer.


Item Type:Article
Related URLs:
URLURL TypeDescription
http://dx.doi.org/10.1088/1742-6596/331/5/052016 DOIUNSPECIFIED
http://iopscience.iop.org/1742-6596/331/5/052016/PublisherUNSPECIFIED
Additional Information:© 2011 Institute of Physics. Published under licence by IOP Publishing Ltd.
Subject Keywords:Accelerators, beams and electromagnetism; Electronics and devices; Nuclear physics; Instrumentation and measurement; Particle physics and field theory
Classification Code:PACS: 84.40.Ua; 07.05.Kf; 07.05.Wr; 29.50.+v; 29.85.-c; 07.05.Bx; 29.40.Gx
Record Number:CaltechAUTHORS:20120410-110725216
Persistent URL:https://resolver.caltech.edu/CaltechAUTHORS:20120410-110725216
Official Citation:High Throughput WAN Data Transfer with Hadoop-based Storage A Amin et al 2011 J. Phys.: Conf. Ser. 331 052016
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:30048
Collection:CaltechAUTHORS
Deposited By: Tony Diaz
Deposited On:10 Apr 2012 21:07
Last Modified:03 Oct 2019 03:46

Repository Staff Only: item control page