SparkSeq: fast, scalable and cloud-ready tool for the interactive genomic data analysis with nucleotide precision.
Many time-consuming analyses of next -: generation sequencing data can be addressed with modern cloud computing. The Apache Hadoop-based solutions have become popular in genomics BECAUSE OF: their scalability in a cloud infrastructure. So far, most of these tools have been used for batch data processing rather than interactive data querying. The SparkSeq software has been created to take advantage of a new MapReduce framework, Apache Spark, for next-generation sequencing data. SparkSeq is a general-purpose, flexible and easily extendable library for genomic cloud computing. It can be used to build genomic analysis pipelines in Scala and run them in an interactive way. SparkSeq opens up the possibility of customized ad hoc secondary analyses and iterative machine learning algorithms. This article demonstrates its scalability and overall fast performance by running the analyses of sequencing datasets. Tests of SparkSeq also prove that the use of cache and HDFS block size can be tuned for the optimal performance on multiple worker nodes.


METHODS
Available under open source Apache 2.0 license: https://bitbucket.org/mwiewiorka/sparkseq/.
Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland.
Marek S
MS
Wiewiórka
Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland.
Antonio
A
Messina
Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland.
Alicja
A
Pacholewska
Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland.
Sergio
S
Maffioletti
Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland.
Piotr
P
Gawrysiak
Institute of Computer Science, Warsaw University of Technology, Warsaw, Poland, ICS 00-665 Warsaw (MW, PG), Grid Computing Competence Center-GC3, University of Zurich, 8057 Zürich (SM, AM), Swiss Institute of Equine Medicine, Vetsuisse Faculty, University of Bern and ALP-Haras, 3001 Bern (AP), Institute of Genetics, Vetsuisse Faculty, University of Bern, Bern, 3001 Bern (AP) and Functional Genomics Center Zurich, CH-8057 Zurich, Switzerland.
Michał J
MJ
Okoniewski
D000465:Algorithms; D023281:Genomics / Q000379:methods*; D059014:High-Throughput Nucleotide Sequencing / Q000379:methods*; D020407:Internet; D009711:Nucleotides / Q000235:genetics*; D012984:Software; D013223:Statistics as Topic / Q000379:methods*; D013997:Time Factors
D009711:Nucleotides