title: Low concordance of multiple variant-calling pipelines: practical implications for exome and genome sequencing.
abstract: BACKGROUND
  To facilitate the clinical implementation of genomic medicine by next-generation sequencing, it will be critically important to obtain accurate and consistent variant calls on personal genomes. Multiple software tools for variant calling are available, but it is unclear how comparable these tools are or what their relative merits in real-world scenarios might be.
  
  
  METHODS
  We sequenced 15 exomes from four families using commercial kits (Illumina HiSeq 2000 platform and Agilent SureSelect version 2 capture kit), with approximately 120X mean coverage. We analyzed the raw data using near-default parameters with five different alignment and variant-calling pipelines (SOAP, BWA-GATK, BWA-SNVer, GNUMAP, and BWA-SAMtools). We additionally sequenced a single whole genome using the sequencing and analysis pipeline from Complete Genomics (CG), with 95% of the exome region being covered by 20 or more reads per base. Finally, we validated 919 single-nucleotide variations (SNVs) and 841 insertions and deletions (indels), including similar fractions of GATK-only, SOAP-only, and shared calls, on the MiSeq platform by amplicon sequencing with approximately 5000X mean coverage.
  
  
  RESULTS
  SNV concordance between five Illumina pipelines across all 15 exomes was 57.4%, while 0.5 to 5.1% of variants were called as unique to each pipeline. Indel concordance was only 26.8% between three indel-calling pipelines, even after left-normalizing and intervalizing genomic coordinates by 20 base pairs. There were 11% of CG variants falling within targeted regions in exome sequencing that were not called by any of the Illumina-based exome analysis pipelines. Based on targeted amplicon sequencing on the MiSeq platform, 97.1%, 60.2%, and 99.1% of the GATK-only, SOAP-only and shared SNVs could be validated, but only 54.0%, 44.6%, and 78.1% of the GATK-only, SOAP-only and shared indels could be validated. Additionally, our analysis of two families (one with four individuals and the other with seven), demonstrated additional accuracy gained in variant discovery by having access to genetic data from a multi-generational family.
  
  
  CONCLUSIONS
  Our results suggest that more caution should be exercised in genomic medicine settings when analyzing individual genomes, including interpreting positive and negative findings with scrutiny, especially for indels. We advocate for renewed collection and sequencing of multi-generational families to increase the overall accuracy of whole genomes.
authors.affiliation: Stanley Institute for Cognitive Genomics, Cold Spring Harbor Laboratory, One Bungtown Rd, Cold Spring Harbor, 11724, USA ; Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA.
authors.forename: Jason
authors.initials: J
authors.lastname: O'Rawe
authors.affiliation: BGI-Shenzhen, Shenzhen 518000, China.
authors.forename: Tao
authors.initials: T
authors.lastname: Jiang
authors.affiliation: BGI-Shenzhen, Shenzhen 518000, China.
authors.forename: Guangqing
authors.initials: G
authors.lastname: Sun
authors.affiliation: Stanley Institute for Cognitive Genomics, Cold Spring Harbor Laboratory, One Bungtown Rd, Cold Spring Harbor, 11724, USA ; Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA.
authors.forename: Yiyang
authors.initials: Y
authors.lastname: Wu
authors.affiliation: New Jersey Institute of Technology, Martin Luther King Jr. Blvd, Newark, 07103, USA.
authors.forename: Wei
authors.initials: W
authors.lastname: Wang
authors.affiliation: BGI-Shenzhen, Shenzhen 518000, China.
authors.forename: Jingchu
authors.initials: J
authors.lastname: Hu
authors.affiliation: Brigham Young University, N University Ave, Provo, 84606, USA.
authors.forename: Paul
authors.initials: P
authors.lastname: Bodily
authors.affiliation: Children's Hospital of Philadelphia, Civic Center Blvd, Philadelphia, 19104, USA.
authors.forename: Lifeng
authors.initials: L
authors.lastname: Tian
authors.affiliation: Children's Hospital of Philadelphia, Civic Center Blvd, Philadelphia, 19104, USA.
authors.forename: Hakon
authors.initials: H
authors.lastname: Hakonarson
authors.affiliation: Boston University School of Medicine, E Concord St, Boston, 02118, USA.
authors.forename: W Evan
authors.initials: WE
authors.lastname: Johnson
authors.affiliation: New Jersey Institute of Technology, Martin Luther King Jr. Blvd, Newark, 07103, USA.
authors.forename: Zhi
authors.initials: Z
authors.lastname: Wei
authors.affiliation: University of Southern California, 1501 San Pablo Street, Los Angeles, 90089, USA ; Utah Foundation for Biomedical Research, E 3300 S, Salt Lake City, 84106, USA.
authors.forename: Kai
authors.initials: K
authors.lastname: Wang
authors.affiliation: Stanley Institute for Cognitive Genomics, Cold Spring Harbor Laboratory, One Bungtown Rd, Cold Spring Harbor, 11724, USA ; Stony Brook University, 100 Nicolls Rd, Stony Brook, 11794, USA ; Utah Foundation for Biomedical Research, E 3300 S, Salt Lake City, 84106, USA.
authors.forename: Gholson J
authors.initials: GJ
authors.lastname: Lyon
references.citation: Nature. 2012 Apr 04;485(7397):246-50
references.pmid: 22495309
references.citation: Am J Hum Genet. 2007 Nov;81(5):1084-97
references.pmid: 17924348
references.citation: Nature. 2012 Jul 11;487(7406):190-5
references.pmid: 22785314
references.citation: Nat Biotechnol. 2012 Mar 07;30(3):226-9
references.pmid: 22398614
references.citation: Bioinformatics. 2010 Mar 15;26(6):722-9
references.pmid: 20144947
references.citation: Neuron. 2012 Apr 26;74(2):285-99
references.pmid: 22542183
references.citation: Nature. 2012 Apr 04;485(7397):237-41
references.pmid: 22495306
references.citation: G3 (Bethesda). 2011 Nov;1(6):457-70
references.pmid: 22384356
references.citation: Bioinformatics. 2009 Jul 15;25(14):1754-60
references.pmid: 19451168
references.citation: Nature. 2012 Apr 04;485(7397):242-5
references.pmid: 22495311
references.citation: PLoS Genet. 2012;8(4):e1002635
references.pmid: 22511880
references.citation: Science. 2012 Jul 6;337(6090):100-4
references.pmid: 22604722
references.citation: Nature. 2012 Nov 1;491(7422):56-65
references.pmid: 23128226
references.citation: Nat Genet. 2011 Jun 12;43(7):712-4
references.pmid: 21666693
references.citation: Nature. 2009 Sep 10;461(7261):272-6
references.pmid: 19684571
references.citation: Nat Genet. 2012 Jul 22;44(8):955-9
references.pmid: 22820512
references.citation: Genome Res. 2010 Sep;20(9):1297-303
references.pmid: 20644199
references.citation: Am J Hum Genet. 2012 Aug 10;91(2):238-51
references.pmid: 22883141
references.citation: Genet Med. 2011 Mar;13(3):188-90
references.pmid: 21311341
references.citation: Hum Mol Genet. 2010 Oct 15;19(R2):R131-6
references.pmid: 20858594
references.citation: J Comput Biol. 2012 Mar;19(3):279-92
references.pmid: 22175250
references.citation: Bioinformatics. 2009 Aug 1;25(15):1966-7
references.pmid: 19497933
references.citation: PLoS Genet. 2011 Sep;7(9):e1002280
references.pmid: 21935354
references.citation: PLoS One. 2012;7(7):e40294
references.pmid: 22811759
references.citation: Nat Rev Genet. 2011 Jun;12(6):443-51
references.pmid: 21587300
references.citation: Nucleic Acids Res. 2012 Mar;40(6):2426-31
references.pmid: 22127862
references.citation: Genome Res. 2006 Sep;16(9):1182-90
references.pmid: 16902084
references.citation: Nat Genet. 2012 Dec;44(12):1365-9
references.pmid: 23042115
references.citation: Genomics. 1995 Mar 20;26(2):345-53
references.pmid: 7601461
references.citation: Genome Med. 2012 Jul 26;4(7):58
references.pmid: 22830651
references.citation: Nucleic Acids Res. 2010 Sep;38(16):e164
references.pmid: 20601685
references.citation: Annu Rev Genomics Hum Genet. 2012;13:1-27
references.pmid: 22657391
references.citation: Discov Med. 2011 Jul;12(62):41-55
references.pmid: 21794208
references.citation: Nat Genet. 2011 May;43(5):491-8
references.pmid: 21478889
references.citation: Bioinformatics. 2008 Mar 1;24(5):713-4
references.pmid: 18227114
references.citation: Genome Res. 2011 Jun;21(6):940-51
references.pmid: 21460063
references.citation: Bioinformatics. 2010 Jan 1;26(1):38-45
references.pmid: 19861355
references.citation: Bioinformatics. 2010 Mar 15;26(6):730-6
references.pmid: 20130035
references.citation: Hum Genomics. 2010 Apr;4(4):271-7
references.pmid: 20511140
references.citation: Genome Res. 2010 Feb;20(2):265-72
references.pmid: 20019144
references.citation: Genome Res. 2011 Jun;21(6):830-9
references.pmid: 21460062
references.citation: Genome Res. 2009 Jun;19(6):1124-32
references.pmid: 19420381
references.citation: Nature. 2011 Feb 3;470(7332):59-65
references.pmid: 21293372
references.citation: Science. 2010 Jan 1;327(5961):78-81
references.pmid: 19892942
references.citation: Am J Hum Genet. 2011 Feb 11;88(2):173-82
references.pmid: 21310274
references.citation: Science. 2010 Apr 30;328(5978):636-9
references.pmid: 20220176
references.citation: Nucleic Acids Res. 2011 Oct;39(19):e132
references.pmid: 21813454
references.citation: Brief Bioinform. 2013 Jan;14(1):46-55
references.pmid: 22707752
references.citation: Nat Genet. 2010 Sep;42(9):790-3
references.pmid: 20711175
references.citation: Nat Biotechnol. 2011 Dec 18;30(1):78-82
references.pmid: 22178993
references.citation: Bioinformatics. 2009 Aug 15;25(16):2078-9
references.pmid: 19505943
references.citation: Science. 2012 Jul 6;337(6090):64-9
references.pmid: 22604720
references.citation: Nat Genet. 2010 Jan;42(1):30-5
references.pmid: 19915526
references.citation: Proc Natl Acad Sci U S A. 2012 Jul 24;109(30):11920-7
references.pmid: 22797899
references.citation: Nat Biotechnol. 2011 Dec 18;30(1):61-8
references.pmid: 22178994
references.citation: Am J Hum Genet. 2011 Jul 15;89(1):28-43
references.pmid: 21700266
references.citation: Bioinformatics. 2012 Aug 15;28(16):2097-105
references.pmid: 22668792