Experiences with workflows for automating data-intensive bioinformatics.
High-throughput technologies, such as next-generation sequencing, have turned molecular biology into a data-intensive discipline, requiring bioinformaticians to use high-performance computing resources and carry out data management and analysis tasks on large scale. Workflow systems can be useful to simplify construction of analysis pipelines that automate tasks, support reproducibility and provide measures for fault-tolerance. However, workflow systems can incur significant development and administration overhead so bioinformatics pipelines are often still built without them. We present the experiences with workflows and workflow systems within the bioinformatics community participating in a series of hackathons and workshops of the EU COST action SeqAhead. The organizations are working on similar problems, but we have addressed them with different strategies and solutions. This fragmentation of efforts is inefficient and leads to redundant and incompatible solutions. Based on our experiences we define a set of recommendations for future systems to enable efficient yet simple bioinformatics workflow construction and execution.
Department of Pharmaceutical Biosciences and Science for Life Laboratory, Uppsala University, SE-75124, Uppsala, P.O. Box 591, Sweden. ola.spjuth@farmbio.uu.se.
Ola
O
Spjuth
SLU-Global Bioinformatics Centre, Department of Animal Breeding and Genetics, Swedish University of Agricultural Sciences, Uppsala, Sweden. Erik.Bongcam@slu.se.
Erik
E
Bongcam-Rudloff
Science for Life Laboratory, Karolinska Institutet, SE-17121, Stockholm, P.O. Box 1031, Sweden. guillermo.carrasco@scilifelab.se.
Guillermo Carrasco
GC
Hernández
Division of Genetic Epidemiology, Medical University of Innsbruck, Innsbruck, 6020, Austria. lukas.forer@i-med.ac.at.
Lukas
L
Forer
Science for Life Laboratory, Karolinska Institutet, SE-17121, Stockholm, P.O. Box 1031, Sweden. mario.giovacchini@scilifelab.se.
Mario
M
Giovacchini
Science for Life Laboratory, Karolinska Institutet, SE-17121, Stockholm, P.O. Box 1031, Sweden. brainstorm@nopcode.org.
Roman Valls
RV
Guimera
CSC - IT Center for Science Ltd., FI-02101, Espoo, P.O. Box 405, Finland. aleksi.kallio@csc.fi.
Aleksi
A
Kallio
CSC - IT Center for Science Ltd., FI-02101, Espoo, P.O. Box 405, Finland. eija.korpelainen@csc.fi.
Eija
E
Korpelainen
Chair of Bioinformatics Research Group, Boku University, Vienna, Austria. maciej.kandula@boku.ac.at.
Maciej M
MM
Kańduła
Faculty of Mathematics and Informatics, Sofia University, Sofia, Bulgaria. wfxp@milko.3mhz.net.
Milko
M
Krachunov
Chair of Bioinformatics Research Group, Boku University, Vienna, Austria. david.kreil@boku.ac.at.
David P
DP
Kreil
Faculty of Mathematics and Informatics, Sofia University, Sofia, Bulgaria. okulev@fmi.uni-sofia.bg.
Ognyan
O
Kulev
Chair of Bioinformatics Research Group, Boku University, Vienna, Austria. pawel.labaj@boku.ac.at.
Paweł P
PP
Łabaj
Department of Pharmaceutical Biosciences and Science for Life Laboratory, Uppsala University, SE-75124, Uppsala, P.O. Box 591, Sweden. samuel.lampa@it.uu.se.
Samuel
S
Lampa
CRS4 Polaris, Pula, Italy. luca.pireddu@crs4.it.
Luca
L
Pireddu
Division of Genetic Epidemiology, Medical University of Innsbruck, Innsbruck, 6020, Austria. sebastian.schoenherr@i-med.ac.at.
Sebastian
S
Schönherr
Department of Information Technology, Uppsala University, SE-75105, Uppsala, P.O. Box 337, Sweden. alexey.siretskiy@it.uu.se.
Alexey
A
Siretskiy
AgroBioInstitute and Joint Genomic Centre, Sofia, Bulgaria. jim6329@gmail.com.
Dimitar
D
Vassilev
D019295:Computational Biology / Q000379:methods*; D001330:Electronic Data Processing / Q000379:methods*; D059014:High-Throughput Nucleotide Sequencing; D015203:Reproducibility of Results; D057188:Workflow