title: BEETL-fastq: a searchable compressed archive for DNA reads.
abstract: BACKGROUND
  FASTQ is a standard file format for DNA sequencing data, which stores both nucleotides and quality scores. A typical sequencing study can easily generate hundreds of gigabytes of FASTQ files, while public archives such as ENA and NCBI and large international collaborations such as the Cancer Genome Atlas can accumulate many terabytes of data in this format. Compression tools such as gzip are often used to reduce the storage burden but have the disadvantage that the data must be decompressed before they can be used. Here, we present BEETL-fastq, a tool that not only compresses FASTQ-formatted DNA reads more compactly than gzip but also permits rapid search for k-mer queries within the archived sequences. Importantly, the full FASTQ record of each matching read or read pair is returned, allowing the search results to be piped directly to any of the many standard tools that accept FASTQ data as input.
  
  
  RESULTS
  We show that 6.6 terabytes of human reads in FASTQ format can be transformed into 1.7 terabytes of indexed files, from where we can search for 1, 10, 100, 1000 and a million of 30-mers in 3, 8, 14, 45 and 567 s, respectively, plus 20 ms per output read. Useful applications of the search capability are highlighted, including the genotyping of structural variant breakpoints and 'in silico pull-down' experiments in which only the reads that cover a region of interest are selectively extracted for the purposes of variant calling or visualization.
  
  
  METHODS
  BEETL-fastq is part of the BEETL library, available as a github repository at github.com/BEETL/BEETL.
authors.affiliation: Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.
authors.forename: Lilian
authors.initials: L
authors.lastname: Janin
authors.affiliation: Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.
authors.forename: Ole
authors.initials: O
authors.lastname: Schulz-Trieglaff
authors.affiliation: Computational Biology Group, Illumina Cambridge Ltd., Little Chesterford, Essex CB10 1XL, UK.
authors.forename: Anthony J
authors.initials: AJ
authors.lastname: Cox
mesh_terms: D000465:Algorithms; D003198:Computer Simulation; D004247:DNA; D044962:Data Compression / Q000379:methods*; D016678:Genome; D015894:Genome, Human; D005838:Genotype; D059014:High-Throughput Nucleotide Sequencing; D006801:Humans; D009369:Neoplasms / Q000235:genetics*; D017422:Sequence Analysis, DNA / Q000379:methods*; D012984:Software
chemical_list: D004247:DNA