Maxa, Zdenek and Ahmed, Badar and Kcira, Dorian and Legrand, Iosif and Mughal, Azher and Thomas, Michael and Voicu, Ramiro (2011) Powering physics data transfers with FDT. Journal of Physics: Conference Series, 331 . Art. No. 052014. ISSN 1742-6596 http://resolver.caltech.edu/CaltechAUTHORS:20120411-080550406
- Published Version
See Usage Policy.
Use this Persistent URL to link to this item: http://resolver.caltech.edu/CaltechAUTHORS:20120411-080550406
We present a data transfer system for the grid environment built on top of the open source FDT tool (Fast Data Transfer) developed by Caltech in collaboration with the National University of Science and Technology (Pakistan). The enhancement layer above FDT consists of a client program - fdtcp (FDT copy) and a fdtd service (FDT daemon). This pair of components allows for GSI authenticated data transfers and offers to the user (or data movement production service) interface analogous to grid middle-ware data transfer services - SRM (i.e. srmcp) or GridFTP (i.e. globus-url-copy). fdtcp/fdtd enables third-party, batched file transfers. An important aspect is monitoring by means of the MonALISA active monitoring light-weight library ApMon, providing real-time monitoring and arrival time estimates as well as powerful troubleshooting mechanism. The actual transfer is carried out by the FDT application, an efficient application capable of reading and writing at disk speed over wide area networks. FDT's excellent performance was demonstrated e.g. during SuperComputing 2009 Bandwidth Challenge. We also discuss the storage technology interface layer, specifically focusing on the open source Hadoop distributed file system (HDFS), presenting the recently developed FDT-HDFS sequential write adapter. The integration with CMS PhEDEx is described as well. The PhEDEx project (Physics Experiment Data Export) is responsible for facilitating large-scale CMS data transfers across the grid. Ongoing and future development involves interfacing with next generation network services developed by OGF NSI-WG, GLIF and DICE groups, allowing for network resource reservation and scheduling.
|Additional Information:||© 2011 Institute of Physics. Published under licence by IOP Publishing Ltd. This work was supported by the National Science Foundation within the DISUN grant, contract No. PHY-0533280, by the National Science Foundation within the Ultralight grant, contract No. PHY-0427110.|
|Subject Keywords:||Accelerators, beams and electromagnetism; Electronics and devices; Nuclear physics; Instrumentation and measurement; Particle physics and field theory|
|Classification Code:||PACS: 29.40.Gx; 07.50.Ek; 84.40.Ua; 07.05.Wr; 29.50.+v; 29.85.-c|
|Official Citation:||Powering physics data transfers with FDT Zdenek Maxa et al 2011 J. Phys.: Conf. Ser. 331 052014|
|Usage Policy:||No commercial reproduction, distribution, display or performance rights in this work are provided.|
|Deposited By:||Tony Diaz|
|Deposited On:||11 Apr 2012 16:55|
|Last Modified:||26 Dec 2012 15:02|
Repository Staff Only: item control page