The balance of reproducibility, sensitivity, and specificity of lists of differentially expressed genes in microarray studies.
BACKGROUND
Reproducibility is a fundamental requirement in scientific experiments. Some recent publications have claimed that microarrays are unreliable because lists of differentially expressed genes (DEGs) are not reproducible in similar experiments. Meanwhile, new statistical methods for identifying DEGs continue to appear in the scientific literature. The resultant variety of existing and emerging methods exacerbates confusion and continuing debate in the microarray community on the appropriate choice of methods for identifying reliable DEG lists.


RESULTS
Using the data sets generated by the MicroArray Quality Control (MAQC) project, we investigated the impact on the reproducibility of DEG lists of a few widely used gene selection procedures. We present comprehensive results from inter-site comparisons using the same microarray platform, cross-platform comparisons using multiple microarray platforms, and comparisons between microarray results and those from TaqMan - the widely regarded "standard" gene expression platform. Our results demonstrate that (1) previously reported discordance between DEG lists could simply result from ranking and selecting DEGs solely by statistical significance (P) derived from widely used simple t-tests; (2) when fold change (FC) is used as the ranking criterion with a non-stringent P-value cutoff filtering, the DEG lists become much more reproducible, especially when fewer genes are selected as differentially expressed, as is the case in most microarray studies; and (3) the instability of short DEG lists solely based on P-value ranking is an expected mathematical consequence of the high variability of the t-values; the more stringent the P-value threshold, the less reproducible the DEG list is. These observations are also consistent with results from extensive simulation calculations.


CONCLUSIONS
We recommend the use of FC-ranking plus a non-stringent P cutoff as a straightforward and baseline practice in order to generate more reproducible DEG lists. Specifically, the P-value cutoff should not be stringent (too small) and FC should be as large as possible. Our results provide practical guidance to choose the appropriate FC and P-value cutoffs when selecting a given number of DEGs. The FC criterion enhances reproducibility, whereas the P criterion balances sensitivity and specificity.
Shi
Leming
L
National Center for Toxicological Research, US Food and Drug Administration, 3900 NCTR Road, Jefferson, AR 72079, USA. leming.shi@fda.hhs.gov
Jones
Wendell D
WD
Jensen
Roderick V
RV
Harris
Stephen C
SC
Perkins
Roger G
RG
Goodsaid
Federico M
FM
Guo
Lei
L
Croner
Lisa J
LJ
Boysen
Cecilie
C
Fang
Hong
H
Qian
Feng
F
Amur
Shashi
S
Bao
Wenjun
W
Barbacioru
Catalin C
CC
Bertholet
Vincent
V
Cao
Xiaoxi Megan
XM
Chu
Tzu-Ming
TM
Collins
Patrick J
PJ
Fan
Xiao-Hui
XH
Frueh
Felix W
FW
Fuscoe
James C
JC
Guo
Xu
X
Han
Jing
J
Herman
Damir
D
Hong
Huixiao
H
Kawasaki
Ernest S
ES
Li
Quan-Zhen
QZ
Luo
Yuling
Y
Ma
Yunqing
Y
Mei
Nan
N
Peterson
Ron L
RL
Puri
Raj K
RK
Shippy
Richard
R
Su
Zhenqiang
Z
Sun
Yongming Andrew
YA
Sun
Hongmei
H
Thorn
Brett
B
Turpaz
Yaron
Y
Wang
Charles
C
Wang
Sue Jane
SJ
Warrington
Janet A
JA
Willey
James C
JC
Wu
Jie
J
Xie
Qian
Q
Zhang
Liang
L
Zhang
Lu
L
Zhong
Sheng
S
Wolfinger
Russell D
RD
Tong
Weida
W
D000465:Algorithms; D003198:Computer Simulation; D003627:Data Interpretation, Statistical; D020869:Gene Expression Profiling / Q000379:methods*; D005796:Genes / Q000235:genetics*; D008957:Models, Genetic; D015233:Models, Statistical; D020411:Oligonucleotide Array Sequence Analysis / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity
Lancet. 2005 Feb 5-11;365(9458):454-5
15705441
Nat Biotechnol. 2004 May;22(5):615-21
15122300
Nature. 2006 Aug 31;442(7106):1067-70
16943838
BMC Genomics. 2006 Mar 21;7:59
16551369
Nat Rev Drug Discov. 2005 May;4(5):362-3
15902768
Biostatistics. 2001 Jun;2(2):183-201
12933549
J Neurosci. 2004 Aug 25;24(34):7445-54
15329391
BMC Genomics. 2004 Sep 02;5:61
15345031
Nat Biotechnol. 2006 Sep;24(9):1123-31
16964226
Science. 2003 Oct 17;302(5644):393; author reply 393
14563990
Nat Biotechnol. 2006 Sep;24(9):1151-61
16964229
Nucleic Acids Res. 2003 Oct 1;31(19):5676-84
14500831
Nat Biotechnol. 2007 Jan;25(1):25-6; author reply 26-7
17211383
Science. 2002 Oct 18;298(5593):601-4
12228721
Bioinformatics. 2005 Jan 15;21(2):171-8
15308542
Genome Biol. 2001;2(12):PREPRINT0009
11790248
Proc Natl Acad Sci U S A. 2006 Jan 17;103(3):649-53
16407153
Pharmacogenomics J. 2007 Jun;7(3):212-20
16940966
Proc Natl Acad Sci U S A. 2002 Aug 20;99(17):11387-92
12177426
Cell. 2006 Nov 17;127(4):657-9
17110319
Science. 2004 Oct 22;306(5696):630-1
15499004
Nucleic Acids Res. 2005 Jan 07;33(1):56-65
15640445
Lancet. 2005 Feb 5-11;365(9458):488-92
15705458
Science. 1995 Oct 20;270(5235):467-70
7569999
Nat Rev Genet. 2006 Jan;7(1):55-65
16369572
Proc Natl Acad Sci U S A. 2002 Nov 12;99(23):15000-5
12415109
Science. 2006 Sep 15;313(5793):1559
16973852
Nucleic Acids Res. 2004 Oct 12;32(18):5471-9
15479783
Biostatistics. 2005 Jan;6(1):59-75
15618528
Nat Biotechnol. 2006 Sep;24(9):1115-22
16964225
Genome Res. 2003 Jul;13(7):1775-85
12805270
Proc Natl Acad Sci U S A. 2006 Apr 11;103(15):5923-8
16585533
Nat Rev Genet. 2004 Mar;5(3):229-37
14970825
Plant Cell. 2004 May;16(5):1314-26
15100403
Bioinformatics. 2001 Jun;17(6):509-19
11395427
Nat Methods. 2005 May;2(5):345-50
15846361
Genome Res. 2000 Dec;10(12):2022-9
11116096
Curr Opin Biotechnol. 2008 Feb;19(1):10-8
18155896
Int J Cardiol. 2005 Jul 10;102(2):333-40
15982506
Proc Natl Acad Sci U S A. 2001 Apr 24;98(9):5116-21
11309499
Nat Biotechnol. 2006 Sep;24(9):1162-9
17061323
Cell. 2000 Jul 7;102(1):109-26
10929718
Nat Biotechnol. 1996 Dec;14(13):1675-80
9634850
Stat Methods Med Res. 2006 Feb;15(1):3-20
16477945
J Comput Biol. 2001;8(1):37-52
11339905
BMC Bioinformatics. 2007 Oct 25;8:412
17961233
Science. 2002 Oct 18;298(5593):597-600
12228720
BMC Bioinformatics. 2005 Jul 15;6 Suppl 2:S12
16026597
Nucleic Acids Res. 2004 May 25;32(9):e74
15161944
Nat Biotechnol. 2006 Jul;24(7):832-40
16823376