Efficient storage of high throughput DNA sequencing data using reference-based compression.
Data storage costs have become an appreciable proportion of total cost in the creation and analysis of DNA sequence data. Of particular concern is that the rate of increase in DNA sequencing is significantly outstripping the rate of increase in disk storage capacity. In this paper we present a new reference-based compression method that efficiently compresses DNA sequences for storage. Our approach works for resequencing experiments that target well-studied genomes. We align new sequences to a reference genome and then encode the differences between the new sequence and the reference genome for storage. Our compression method is most efficient when we allow controlled loss of data in the saving of quality information and unaligned sequences. With this new compression method we observe exponential efficiency gains as read lengths increase, and the magnitude of this efficiency gain can be controlled by changing the amount of quality information stored. Our compression method is tunable: The storage of quality scores and unaligned sequences may be adjusted for different experiments to conserve information or to minimize storage costs, and provides one opportunity to address the threat that increasing DNA sequence volumes will overcome our ability to store the sequences.
European Molecular Biology Laboratory's European Bioinformatics Institute (EMBL-EBI), Wellcome Trust Genome Campus, Hinxton, Cambridgeshire CB10 1SD, United Kingdom.
Markus
M
Hsi-Yang Fritz
Rasko
R
Leinonen
Guy
G
Cochrane
Ewan
E
Birney
D000465:Algorithms; D044962:Data Compression / Q000379:methods*; D023281:Genomics / Q000379:methods*; D059014:High-Throughput Nucleotide Sequencing; D012015:Reference Standards; D017422:Sequence Analysis, DNA / Q000379:methods*; D012984:Software
Bioinformatics. 2009 Jul 15;25(14):1754-60
19451168
Genome Inform Ser Workshop Genome Inform. 2000;11:43-52
11700586
Nature. 2010 Oct 28;467(7319):1061-73
20981092
Nucleic Acids Res. 2010 Jan;38(Database issue):D870-1
19965774
BMC Bioinformatics. 2010 Oct 14;11:514
20946637
Genome Res. 1998 Mar;8(3):186-94
9521922
Nature. 2005 Sep 15;437(7057):376-80
16056220
Proc Natl Acad Sci U S A. 1977 Dec;74(12):5463-7
271968
Bioinformatics. 2009 Jan 15;25(2):274-5
18996942
Bioinformatics. 2009 Aug 15;25(16):2078-9
19505943
Science. 2004 Oct 22;306(5696):636-40
15499007
Nature. 2010 Apr 15;464(7291):993-8
20393554
Genome Biol. 2010;11(5):207
20441614
Bioinformatics. 2002 Dec;18(12):1696-8
12490460