Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to Alzheimer's disease.
The authors evaluated whether there is an excess of statistically significant results in studies of genetic associations with Alzheimer's disease reflecting either between-study heterogeneity or bias. Among published articles on genetic associations entered into the comprehensive AlzGene database (www.alzgene.org) through January 31, 2007, 1,348 studies included in 175 meta-analyses with 3 or more studies each were analyzed. The number of observed studies (O) with statistically significant results (P = 0.05 threshold) was compared with the expected number (E) under different assumptions for the magnitude of the effect size. In the main analysis, the plausible effect size of each association was the summary effect presented in the respective meta-analysis. Overall, 19 meta-analyses (all with eventually nonsignificant summary effects) had a documented excess of O over E: Typically single studies had significant effects pointing in opposite directions and early summary effects were dissipated over time. Across the whole domain, O was 235 (17.4%), while E was 164.8 (12.2%) (P < 10(-6)). The excess showed a predilection for meta-analyses with nonsignificant summary effects and between-study heterogeneity. The excess was seen for all levels of statistical significance and also for studies with borderline P values (P = 0.05-0.10). The excess of significant findings may represent significance-chasing biases in a setting of massive testing.
Kavvoura
Fotini K
FK
Department of Hygiene and Epidemiology, University of Ioannina School of Medicine, Ioannina, Greece.
McQueen
Matthew B
MB
Khoury
Muin J
MJ
Tanzi
Rudolph E
RE
Bertram
Lars
L
Ioannidis
John P A
JP
D000544:Alzheimer Disease / Q000453:epidemiology* / Q000235:genetics*; D015982:Bias; D020022:Genetic Predisposition to Disease / Q000453:epidemiology*; D006801:Humans
Stat Med. 2002 Jun 15;21(11):1539-58
12111919
Nat Genet. 2001 Nov;29(3):306-9
11600885
JAMA. 1992 Jan 15;267(3):374-8
1727960
Nature. 2007 Jun 28;447(7148):1087-93
17529967
Am J Prev Med. 2007 Oct;33(4):310-7
17888858
PLoS One. 2007 Sep 05;2(9):e841
17786212
Lancet. 1991 Apr 13;337(8746):867-72
1672966
JAMA. 2004 May 26;291(20):2457-65
15161896
CMAJ. 2004 Sep 28;171(7):735-40
15451835
Am J Epidemiol. 2007 May 1;165(9):973-84
17293603
J Natl Cancer Inst. 1959 Apr;22(4):719-48
13655060
Control Clin Trials. 1986 Sep;7(3):177-88
3802833
PLoS Biol. 2005 Sep;3(9):e327
16149852
Cancer Epidemiol Biomarkers Prev. 2006 Apr;15(4):595-8
16614096
PLoS Med. 2005 Aug;2(8):e124
16060722
J Clin Epidemiol. 2005 Jun;58(6):543-9
15878467
Hum Hered. 2007;64(4):203-13
17551261
Am J Epidemiol. 2005 Aug 15;162(4):302-4
16014777
Biometrics. 1994 Dec;50(4):1088-101
7786990
Genet Med. 2002 Mar-Apr;4(2):45-61
11882781
Trends Genet. 2006 Jul;22(7):350-4
16713652
Clin Trials. 2007;4(3):245-53
17715249
Stat Med. 2001 Feb 28;20(4):641-54
11223905
PLoS Med. 2006 Dec;3(12):e520
17194198
Control Clin Trials. 2000 Oct;21(5):415-27
11018560
Mol Psychiatry. 2007 May;12(5):454-61
17453061
BMJ. 2006 Sep 16;333(7568):597-600
16974018
JAMA. 1990 Mar 9;263(10):1385-9
2406472
J Clin Epidemiol. 2006 Dec;59(12):1312-8
17098574
BMJ. 1997 Sep 13;315(7109):629-34
9310563
Nat Genet. 2005 Apr;37(4):413-7
15793588
Psychol Methods. 2006 Jun;11(2):193-206
16784338
Am J Hum Genet. 2007 Mar;80(3):531-8
17273975
Int J Epidemiol. 2008 Feb;37(1):120-32
17898028
BMJ. 2005 Jul 30;331(7511):267-70
16052019
JAMA. 1998 Jan 28;279(4):281-6
9450711
PLoS Med. 2007 Mar;4(3):e79
17341129
PLoS Med. 2005 Dec;2(12):e334
16285839
Nat Genet. 2006 Jan;38(1):3-5
16468121
Am J Hum Genet. 2007 Apr;80(4):605-15
17357068
Nat Genet. 2007 Jan;39(1):17-23
17192785
Contemp Clin Trials. 2005 Apr;26(2):244-51
15837444
Ann N Y Acad Sci. 1993 Dec 31;703:135-46; discussion 146-8
8192291
Nature. 2007 Jun 7;447(7145):655-60
17554299
J Clin Epidemiol. 2000 Nov;53(11):1119-29
11106885
Nat Genet. 2007 Sep;39(9):1045-51
17728769
Science. 2007 Jun 1;316(5829):1336-41
17463249
BMJ. 2003 Sep 6;327(7414):557-60
12958120