A Caltech Library Service

Fragment assignment in the cloud with eXpress-D

Roberts, Adam and Feng, Harvey and Pachter, Lior (2013) Fragment assignment in the cloud with eXpress-D. BMC Bioinformatics, 14 . Art. No. 358. ISSN 1471-2105. PMCID PMC3881492. doi:10.1186/1471-2105-14-358.

[img] PDF - Published Version
Creative Commons Attribution.

[img] Archive (ZIP) (Additional file 1: This script contains procedures for repeating our analyses in comparing RSEM , Cufflinks , and eXpress) - Supplemental Material
Creative Commons Attribution.

[img] PDF (Authors’ original file for figure 1) - Supplemental Material
Creative Commons Attribution.

[img] PDF (Authors’ original file for figure 2) - Supplemental Material
Creative Commons Attribution.

[img] PDF (Authors’ original file for figure 3) - Supplemental Material
Creative Commons Attribution.


Use this Persistent URL to link to this item:


Background: Probabilistic assignment of ambiguously mapped fragments produced by high-throughput sequencing experiments has been demonstrated to greatly improve accuracy in the analysis of RNA-Seq and ChIP-Seq, and is an essential step in many other sequence census experiments. A maximum likelihood method using the expectation-maximization (EM) algorithm for optimization is commonly used to solve this problem. However, batch EM-based approaches do not scale well with the size of sequencing datasets, which have been increasing dramatically over the past few years. Thus, current approaches to fragment assignment rely on heuristics or approximations for tractability. Results: We present an implementation of a distributed EM solution to the fragment assignment problem using Spark, a data analytics framework that can scale by leveraging compute clusters within datacenters–“the cloud”. We demonstrate that our implementation easily scales to billions of sequenced fragments, while providing the exact maximum likelihood assignment of ambiguous fragments. The accuracy of the method is shown to be an improvement over the most widely used tools available and can be run in a constant amount of time when cluster resources are scaled linearly with the amount of input data. Conclusions: The cloud offers one solution for the difficulties faced in the analysis of massive high-thoughput sequencing data, which continue to grow rapidly. Researchers in bioinformatics must follow developments in distributed systems–such as new frameworks like Spark–for ways to port existing methods to the cloud and help them scale to the datasets of the future. Our software, eXpress-D, is freely available at:

Item Type:Article
Related URLs:
URLURL TypeDescription CentralArticle
Pachter, Lior0000-0002-9164-6231
Additional Information:© Roberts et al.; licensee BioMed Central Ltd. 2013. This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Received: 13 September 2013. Accepted: 18 November 2013. Published: 7 December 2013. We thank Matei Zaharia, Kristal Curtis, and Reynold Xin for discussions on the feasibility of the Spark implementation. Adam Roberts was supported by an NSF graduate research fellowship. Lior Pachter was partially supported by NIH HG006129. Authors’ contributions: AR developed the method. AR and HF implemented the method and analyzed the results. AR, HF, and LP wrote the manuscript. All authors read and approved the final manuscript. Availability and usage: eXpress-D and Spark are open source software that can be downloaded from their respective websites, and For ease of use, the eXpress-D source code includes a copy of a Spark script that allow users to launch, setup and manage EC2 clusters running Spark and HDFS. The script can be used to launch all nodes in the cluster using a customized Amazon Machine Image (AMI)-a type of templated operating system [24]-that is preloaded with eXpress-D source and binaries. Target and fragment datasets can then be loaded into HDFS or S3 for distributed execution. The eXpress-D wiki page includes more detail about using the script to launch clusters, as well as notes on cluster configuration and tuning. The authors declare that they have no competing interest.
Funding AgencyGrant Number
NSF Graduate Research FellowshipUNSPECIFIED
PubMed Central ID:PMC3881492
Record Number:CaltechAUTHORS:20170303-144424899
Persistent URL:
Usage Policy:No commercial reproduction, distribution, display or performance rights in this work are provided.
ID Code:74725
Deposited By: George Porter
Deposited On:04 Mar 2017 00:13
Last Modified:11 Nov 2021 05:29

Repository Staff Only: item control page