title: A quantitative assessment of the Hadoop framework for analyzing massively parallel DNA sequencing data.
abstract: BACKGROUND
  New high-throughput technologies, such as massively parallel sequencing, have transformed the life sciences into a data-intensive field. The most common e-infrastructure for analyzing this data consists of batch systems that are based on high-performance computing resources; however, the bioinformatics software that is built on this platform does not scale well in the general case. Recently, the Hadoop platform has emerged as an interesting option to address the challenges of increasingly large datasets with distributed storage, distributed processing, built-in data locality, fault tolerance, and an appealing programming methodology.
  
  
  RESULTS
  In this work we introduce metrics and report on a quantitative comparison between Hadoop and a single node of conventional high-performance computing resources for the tasks of short read mapping and variant calling. We calculate efficiency as a function of data size and observe that the Hadoop platform is more efficient for biologically relevant data sizes in terms of computing hours for both split and un-split data files. We also quantify the advantages of the data locality provided by Hadoop for NGS problems, and show that a classical architecture with network-attached storage will not scale when computing resources increase in numbers. Measurements were performed using ten datasets of different sizes, up to 100 gigabases, using the pipeline implemented in Crossbow. To make a fair comparison, we implemented an improved preprocessor for Hadoop with better performance for splittable data files. For improved usability, we implemented a graphical user interface for Crossbow in a private cloud environment using the CloudGene platform. All of the code and data in this study are freely available as open source in public repositories.
  
  
  CONCLUSIONS
  From our experiments we can conclude that the improved Hadoop pipeline scales better than the same pipeline on high-performance computing resources, we also conclude that Hadoop is an economically viable option for the common data sizes that are currently used in massively parallel sequencing. Given that datasets are expected to increase over time, Hadoop is a framework that we envision will have an increasingly important role in future biological data analysis.
authors.affiliation: Department of Information Technology, Uppsala University, P.O. Box 337, Uppsala, SE-75105 Sweden.
authors.forename: Alexey
authors.initials: A
authors.lastname: Siretskiy
authors.affiliation: Department of Information Technology, Uppsala University, P.O. Box 337, Uppsala, SE-75105 Sweden.
authors.forename: Tore
authors.initials: T
authors.lastname: Sundqvist
authors.affiliation: Department of Physical Chemistry, institute of Chemistry, St-Petersburg State University, Saint-Petersburg, Russia.
authors.forename: Mikhail
authors.initials: M
authors.lastname: Voznesenskiy
authors.affiliation: Department of Pharmaceutical Biosciences and Science for Life Laboratory, Uppsala University, P.O. Box 541, Uppsala, SE-75124 Sweden.
authors.forename: Ola
authors.initials: O
authors.lastname: Spjuth
mesh_terms: D019295:Computational Biology; D020407:Internet; D017422:Sequence Analysis, DNA / Q000379:methods*; D012984:Software
keywords: Bioinformatics; DNA-seq; Hadoop; High-performance computing; Massively parallel sequencing; Next generation sequencing