title: BioPig: a Hadoop-based analytic toolkit for large-scale sequence data.
abstract: BACKGROUND
  The recent revolution in sequencing technologies has led to an exponential growth of sequence data. As a result, most of the current bioinformatics tools become obsolete as they fail to scale with data. To tackle this 'data deluge', here we introduce the BioPig sequence analysis toolkit as one of the solutions that scale to data and computation.
  
  
  RESULTS
  We built BioPig on the Apache's Hadoop MapReduce system and the Pig data flow language. Compared with traditional serial and MPI-based algorithms, BioPig has three major advantages: first, BioPig's programmability greatly reduces development time for parallel bioinformatics applications; second, testing BioPig with up to 500 Gb sequences demonstrates that it scales automatically with size of data; and finally, BioPig can be ported without modification on many Hadoop infrastructures, as tested with Magellan system at National Energy Research Scientific Computing Center and the Amazon Elastic Compute Cloud. In summary, BioPig represents a novel program framework with the potential to greatly accelerate data-intensive bioinformatics analysis.
authors.affiliation: Department of Energy, Joint Genome Institute, Walnut Creek, CA 94598, USA and Genomics Division, Lawrence Berkeley National Laboratory, Berkeley, CA 94720, USA.
authors.forename: Henrik
authors.initials: H
authors.lastname: Nordberg
authors.forename: Karan
authors.initials: K
authors.lastname: Bhatia
authors.forename: Kai
authors.initials: K
authors.lastname: Wang
authors.forename: Zhong
authors.initials: Z
authors.lastname: Wang
mesh_terms: D000465:Algorithms; D019295:Computational Biology / Q000379:methods*; D003627:Data Interpretation, Statistical; D059014:High-Throughput Nucleotide Sequencing; D012984:Software