Leveraging the power of high performance computing for next generation sequencing data analysis: tricks and twists from a high throughput exome workflow.
Next generation sequencing (NGS) has been a great success and is now a standard method of research in the life sciences. With this technology, dozens of whole genomes or hundreds of exomes can be sequenced in rather short time, producing huge amounts of data. Complex bioinformatics analyses are required to turn these data into scientific findings. In order to run these analyses fast, automated workflows implemented on high performance computers are state of the art. While providing sufficient compute power and storage to meet the NGS data challenge, high performance computing (HPC) systems require special care when utilized for high throughput processing. This is especially true if the HPC system is shared by different users. Here, stability, robustness and maintainability are as important for automated workflows as speed and throughput. To achieve all of these aims, dedicated solutions have to be developed. In this paper, we present the tricks and twists that we utilized in the implementation of our exome data processing workflow. It may serve as a guideline for other high throughput data analysis projects using a similar infrastructure. The code implementing our solutions is provided in the supporting information files.
Cologne Center for Genomics, University of Cologne, Cologne, Germany.
Amit
A
Kawalia
Cologne Center for Genomics, University of Cologne, Cologne, Germany.
Susanne
S
Motameny
Regional Computing Center Cologne, University of Cologne, Cologne, Germany.
Stephan
S
Wonczak
Cologne Center for Genomics, University of Cologne, Cologne, Germany.
Holger
H
Thiele
Regional Computing Center Cologne, University of Cologne, Cologne, Germany.
Lech
L
Nieroda
Cologne Center for Genomics, University of Cologne, Cologne, Germany.
Kamel
K
Jabbari
Regional Computing Center Cologne, University of Cologne, Cologne, Germany.
Stefan
S
Borowski
Cologne Center for Genomics, University of Cologne, Cologne, Germany.
Vishal
V
Sinha
Cologne Center for Genomics, University of Cologne, Cologne, Germany.
Wilfried
W
Gunia
Regional Computing Center Cologne, University of Cologne, Cologne, Germany.
Ulrich
U
Lang
Regional Computing Center Cologne, University of Cologne, Cologne, Germany.
Viktor
V
Achter
Cologne Center for Genomics, University of Cologne, Cologne, Germany.
Peter
P
Nürnberg
D019295:Computational Biology / Q000379:methods*; D003205:Computing Methodologies; D001330:Electronic Data Processing / Q000379:methods; D059014:High-Throughput Nucleotide Sequencing / Q000379:methods*; D006801:Humans; D017422:Sequence Analysis, DNA / Q000379:methods*; D057188:Workflow