title: Verification and validation of bioinformatics software without a gold standard: a case study of BWA and Bowtie.
abstract: BACKGROUND
  Bioinformatics software quality assurance is essential in genomic medicine. Systematic verification and validation of bioinformatics software is difficult because it is often not possible to obtain a realistic "gold standard" for systematic evaluation. Here we apply a technique that originates from the software testing literature, namely Metamorphic Testing (MT), to systematically test three widely used short-read sequence alignment programs.
  
  
  RESULTS
  MT alleviates the problems associated with the lack of gold standard by checking that the results from multiple executions of a program satisfy a set of expected or desirable properties that can be derived from the software specification or user expectations. We tested BWA, Bowtie and Bowtie2 using simulated data and one HapMap dataset. It is interesting to observe that multiple executions of the same aligner using slightly modified input FASTQ sequence file, such as after randomly re-ordering of the reads, may affect alignment results. Furthermore, we found that the list of variant calls can be affected unless strict quality control is applied during variant calling.
  
  
  CONCLUSIONS
  Thorough testing of bioinformatics software is important in delivering clinical genomic medicine. This paper demonstrates a different framework to test a program that involves checking its properties, thus greatly expanding the number and repertoire of test cases we can apply in practice.
authors.forename: Eleni
authors.initials: E
authors.lastname: Giannoulatou
authors.forename: Shin-Ho
authors.initials: SH
authors.lastname: Park
authors.forename: David T
authors.initials: DT
authors.lastname: Humphreys
authors.forename: Joshua W K
authors.initials: JW
authors.lastname: Ho
mesh_terms: D019295:Computational Biology / Q000379:methods*; D015894:Genome, Human; D023281:Genomics / Q000379:methods*; D059014:High-Throughput Nucleotide Sequencing / Q000379:methods*; D006801:Humans; D016415:Sequence Alignment / Q000379:methods*; D017422:Sequence Analysis, DNA / Q000379:methods*; D012984:Software / Q000592:standards*