title: The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies.
abstract: BACKGROUND
  Reproducibility is a fundamental requirement in scientific experiments. Some recent publications have claimed that microarrays are unreliable because lists of differentially expressed genes (DEGs) are not reproducible in similar experiments. Meanwhile, new statistical methods for identifying DEGs continue to appear in the scientific literature. The resultant variety of existing and emerging methods exacerbates confusion and continuing debate in the microarray community on the appropriate choice of methods for identifying reliable DEG lists.
  
  
  RESULTS
  Using the data sets generated by the MicroArray Quality Control (MAQC) project, we investigated the impact on the reproducibility of DEG lists of a few widely used gene selection procedures. We present comprehensive results from inter-site comparisons using the same microarray platform, cross-platform comparisons using multiple microarray platforms, and comparisons between microarray results and those from TaqMan - the widely regarded "standard" gene expression platform. Our results demonstrate that (1) previously reported discordance between DEG lists could simply result from ranking and selecting DEGs solely by statistical significance (P) derived from widely used simple t-tests; (2) when fold change (FC) is used as the ranking criterion with a non-stringent P-value cutoff filtering, the DEG lists become much more reproducible, especially when fewer genes are selected as differentially expressed, as is the case in most microarray studies; and (3) the instability of short DEG lists solely based on P-value ranking is an expected mathematical consequence of the high variability of the t-values; the more stringent the P-value threshold, the less reproducible the DEG list is. These observations are also consistent with results from extensive simulation calculations.
  
  
  CONCLUSIONS
  We recommend the use of FC-ranking plus a non-stringent P cutoff as a straightforward and baseline practice in order to generate more reproducible DEG lists. Specifically, the P-value cutoff should not be stringent (too small) and FC should be as large as possible. Our results provide practical guidance to choose the appropriate FC and P-value cutoffs when selecting a given number of DEGs. The FC criterion enhances reproducibility, whereas the P criterion balances sensitivity and specificity.
authors.affiliation: National Center for Toxicological Research, US Food and Drug Administration, 3900 NCTR Road, Jefferson, AR 72079, USA. leming.shi@fda.hhs.gov
authors.forename: Leming
authors.initials: L
authors.lastname: Shi
authors.forename: Wendell D
authors.initials: WD
authors.lastname: Jones
authors.forename: Roderick V
authors.initials: RV
authors.lastname: Jensen
authors.forename: Stephen C
authors.initials: SC
authors.lastname: Harris
authors.forename: Roger G
authors.initials: RG
authors.lastname: Perkins
authors.forename: Federico M
authors.initials: FM
authors.lastname: Goodsaid
authors.forename: Lei
authors.initials: L
authors.lastname: Guo
authors.forename: Lisa J
authors.initials: LJ
authors.lastname: Croner
authors.forename: Cecilie
authors.initials: C
authors.lastname: Boysen
authors.forename: Hong
authors.initials: H
authors.lastname: Fang
authors.forename: Feng
authors.initials: F
authors.lastname: Qian
authors.forename: Shashi
authors.initials: S
authors.lastname: Amur
authors.forename: Wenjun
authors.initials: W
authors.lastname: Bao
authors.forename: Catalin C
authors.initials: CC
authors.lastname: Barbacioru
authors.forename: Vincent
authors.initials: V
authors.lastname: Bertholet
authors.forename: Xiaoxi Megan
authors.initials: XM
authors.lastname: Cao
authors.forename: Tzu-Ming
authors.initials: TM
authors.lastname: Chu
authors.forename: Patrick J
authors.initials: PJ
authors.lastname: Collins
authors.forename: Xiao-Hui
authors.initials: XH
authors.lastname: Fan
authors.forename: Felix W
authors.initials: FW
authors.lastname: Frueh
authors.forename: James C
authors.initials: JC
authors.lastname: Fuscoe
authors.forename: Xu
authors.initials: X
authors.lastname: Guo
authors.forename: Jing
authors.initials: J
authors.lastname: Han
authors.forename: Damir
authors.initials: D
authors.lastname: Herman
authors.forename: Huixiao
authors.initials: H
authors.lastname: Hong
authors.forename: Ernest S
authors.initials: ES
authors.lastname: Kawasaki
authors.forename: Quan-Zhen
authors.initials: QZ
authors.lastname: Li
authors.forename: Yuling
authors.initials: Y
authors.lastname: Luo
authors.forename: Yunqing
authors.initials: Y
authors.lastname: Ma
authors.forename: Nan
authors.initials: N
authors.lastname: Mei
authors.forename: Ron L
authors.initials: RL
authors.lastname: Peterson
authors.forename: Raj K
authors.initials: RK
authors.lastname: Puri
authors.forename: Richard
authors.initials: R
authors.lastname: Shippy
authors.forename: Zhenqiang
authors.initials: Z
authors.lastname: Su
authors.forename: Yongming Andrew
authors.initials: YA
authors.lastname: Sun
authors.forename: Hongmei
authors.initials: H
authors.lastname: Sun
authors.forename: Brett
authors.initials: B
authors.lastname: Thorn
authors.forename: Yaron
authors.initials: Y
authors.lastname: Turpaz
authors.forename: Charles
authors.initials: C
authors.lastname: Wang
authors.forename: Sue Jane
authors.initials: SJ
authors.lastname: Wang
authors.forename: Janet A
authors.initials: JA
authors.lastname: Warrington
authors.forename: James C
authors.initials: JC
authors.lastname: Willey
authors.forename: Jie
authors.initials: J
authors.lastname: Wu
authors.forename: Qian
authors.initials: Q
authors.lastname: Xie
authors.forename: Liang
authors.initials: L
authors.lastname: Zhang
authors.forename: Lu
authors.initials: L
authors.lastname: Zhang
authors.forename: Sheng
authors.initials: S
authors.lastname: Zhong
authors.forename: Russell D
authors.initials: RD
authors.lastname: Wolfinger
authors.forename: Weida
authors.initials: W
authors.lastname: Tong
mesh_terms: D000465:Algorithms; D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D020869:Gene Expression Profiling / Q000379:methods*; D005796:Genes / Q000235:genetics*; D008957:Models, Genetic; D015233:Models, Statistical; D020411:Oligonucleotide Array Sequence Analysis / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity
references.citation: Lancet. 2005 Feb 5-11;365(9458):454-5
references.pmid: 15705441
references.citation: Nat Biotechnol. 2004 May;22(5):615-21
references.pmid: 15122300
references.citation: Nature. 2006 Aug 31;442(7106):1067-70
references.pmid: 16943838
references.citation: BMC Genomics. 2006 Mar 21;7:59
references.pmid: 16551369
references.citation: Nat Rev Drug Discov. 2005 May;4(5):362-3
references.pmid: 15902768
references.citation: Biostatistics. 2001 Jun;2(2):183-201
references.pmid: 12933549
references.citation: J Neurosci. 2004 Aug 25;24(34):7445-54
references.pmid: 15329391
references.citation: BMC Genomics. 2004 Sep 02;5:61
references.pmid: 15345031
references.citation: Nat Biotechnol. 2006 Sep;24(9):1123-31
references.pmid: 16964226
references.citation: Science. 2003 Oct 17;302(5644):393; author reply 393
references.pmid: 14563990
references.citation: Nat Biotechnol. 2006 Sep;24(9):1151-61
references.pmid: 16964229
references.citation: Nucleic Acids Res. 2003 Oct 1;31(19):5676-84
references.pmid: 14500831
references.citation: Nat Biotechnol. 2007 Jan;25(1):25-6; author reply 26-7
references.pmid: 17211383
references.citation: Science. 2002 Oct 18;298(5593):601-4
references.pmid: 12228721
references.citation: Bioinformatics. 2005 Jan 15;21(2):171-8
references.pmid: 15308542
references.citation: Genome Biol. 2001;2(12):PREPRINT0009
references.pmid: 11790248
references.citation: Proc Natl Acad Sci U S A. 2006 Jan 17;103(3):649-53
references.pmid: 16407153
references.citation: Pharmacogenomics J. 2007 Jun;7(3):212-20
references.pmid: 16940966
references.citation: Proc Natl Acad Sci U S A. 2002 Aug 20;99(17):11387-92
references.pmid: 12177426
references.citation: Cell. 2006 Nov 17;127(4):657-9
references.pmid: 17110319
references.citation: Science. 2004 Oct 22;306(5696):630-1
references.pmid: 15499004
references.citation: Nucleic Acids Res. 2005 Jan 07;33(1):56-65
references.pmid: 15640445
references.citation: Lancet. 2005 Feb 5-11;365(9458):488-92
references.pmid: 15705458
references.citation: Science. 1995 Oct 20;270(5235):467-70
references.pmid: 7569999
references.citation: Nat Rev Genet. 2006 Jan;7(1):55-65
references.pmid: 16369572
references.citation: Proc Natl Acad Sci U S A. 2002 Nov 12;99(23):15000-5
references.pmid: 12415109
references.citation: Science. 2006 Sep 15;313(5793):1559
references.pmid: 16973852
references.citation: Nucleic Acids Res. 2004 Oct 12;32(18):5471-9
references.pmid: 15479783
references.citation: Biostatistics. 2005 Jan;6(1):59-75
references.pmid: 15618528
references.citation: Nat Biotechnol. 2006 Sep;24(9):1115-22
references.pmid: 16964225
references.citation: Genome Res. 2003 Jul;13(7):1775-85
references.pmid: 12805270
references.citation: Proc Natl Acad Sci U S A. 2006 Apr 11;103(15):5923-8
references.pmid: 16585533
references.citation: Nat Rev Genet. 2004 Mar;5(3):229-37
references.pmid: 14970825
references.citation: Plant Cell. 2004 May;16(5):1314-26
references.pmid: 15100403
references.citation: Bioinformatics. 2001 Jun;17(6):509-19
references.pmid: 11395427
references.citation: Nat Methods. 2005 May;2(5):345-50
references.pmid: 15846361
references.citation: Genome Res. 2000 Dec;10(12):2022-9
references.pmid: 11116096
references.citation: Curr Opin Biotechnol. 2008 Feb;19(1):10-8
references.pmid: 18155896
references.citation: Int J Cardiol. 2005 Jul 10;102(2):333-40
references.pmid: 15982506
references.citation: Proc Natl Acad Sci U S A. 2001 Apr 24;98(9):5116-21
references.pmid: 11309499
references.citation: Nat Biotechnol. 2006 Sep;24(9):1162-9
references.pmid: 17061323
references.citation: Cell. 2000 Jul 7;102(1):109-26
references.pmid: 10929718
references.citation: Nat Biotechnol. 1996 Dec;14(13):1675-80
references.pmid: 9634850
references.citation: Stat Methods Med Res. 2006 Feb;15(1):3-20
references.pmid: 16477945
references.citation: J Comput Biol. 2001;8(1):37-52
references.pmid: 11339905
references.citation: BMC Bioinformatics. 2007 Oct 25;8:412
references.pmid: 17961233
references.citation: Science. 2002 Oct 18;298(5593):597-600
references.pmid: 12228720
references.citation: BMC Bioinformatics. 2005 Jul 15;6 Suppl 2:S12
references.pmid: 16026597
references.citation: Nucleic Acids Res. 2004 May 25;32(9):e74
references.pmid: 15161944
references.citation: Nat Biotechnol. 2006 Jul;24(7):832-40
references.pmid: 16823376