title: Interpreting observational studies: why empirical calibration is needed to correct p-values.
abstract: Often the literature makes assertions of medical product effects on the basis of ' p < 0.05'. The underlying premise is that at this threshold, there is only a 5% probability that the observed effect would be seen by chance when in reality there is no effect. In observational studies, much more than in randomized trials, bias and confounding may undermine this premise. To test this premise, we selected three exemplar drug safety studies from literature, representing a case-control, a cohort, and a self-controlled case series design. We attempted to replicate these studies as best we could for the drugs studied in the original articles. Next, we applied the same three designs to sets of negative controls: drugs that are not believed to cause the outcome of interest. We observed how often p < 0.05 when the null hypothesis is true, and we fitted distributions to the effect estimates. Using these distributions, we compute calibrated p-values that reflect the probability of observing the effect estimate under the null hypothesis, taking both random and systematic error into account. An automated analysis of scientific literature was performed to evaluate the potential impact of such a calibration. Our experiment provides evidence that the majority of observational studies would declare statistical significance when no effect is present. Empirical calibration was found to reduce spurious results to the desired 5% level. Applying these adjustments to literature suggests that at least 54% of findings with p < 0.05 are not actually statistically significant and should be reevaluated.
authors.affiliation: Department of Medical Informatics, Erasmus University Medical Center Rotterdam, Rotterdam, The Netherlands; Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD, U.S.A.
authors.forename: Martijn J
authors.initials: MJ
authors.lastname: Schuemie
authors.forename: Patrick B
authors.initials: PB
authors.lastname: Ryan
authors.forename: William
authors.initials: W
authors.lastname: DuMouchel
authors.forename: Marc A
authors.initials: MA
authors.lastname: Suchard
authors.forename: David
authors.initials: D
authors.lastname: Madigan
mesh_terms: D015982:Bias; D056486:Chemical and Drug Induced Liver Injury / Q000209:etiology; D003627:Data Interpretation, Statistical; D005260:Female; D006471:Gastrointestinal Hemorrhage / Q000209:etiology; D006801:Humans; D007538:Isoniazid / Q000009:adverse effects; D008297:Male; D064887:Observational Studies as Topic / Q000379:methods*; D012107:Research Design; D017367:Selective Serotonin Reuptake Inhibitors / Q000009:adverse effects
chemical_list: D007538:Isoniazid; D017367:Selective Serotonin Reuptake Inhibitors
keywords: calibration; hypothesis testing; negative controls; observational studies
references.citation: Int J Environ Res Public Health. 2010 Apr;7(4):1520-39
references.pmid: 20617044
references.citation: Arch Intern Med. 2003 Mar 24;163(6):728-34
references.pmid: 12639207
references.citation: PLoS Med. 2005 Aug;2(8):e124
references.pmid: 16060722
references.citation: Epidemiology. 2010 May;21(3):383-8
references.pmid: 20335814
references.citation: Pharmacoepidemiol Drug Saf. 2006 Jul;15(7):444-53
references.pmid: 16761308
references.citation: Int J Epidemiol. 1988 Sep;17(3):680-5
references.pmid: 3272133
references.citation: Am J Epidemiol. 1996 Jun 1;143(11):1165-73
references.pmid: 8633607
references.citation: BMJ. 2010 Sep 01;341:c4444
references.pmid: 20813820
references.citation: J Clin Epidemiol. 2002 Mar;55(3):310-3
references.pmid: 11864803
references.citation: BMC Bioinformatics. 2008 Jul 09;9:303
references.pmid: 18613966
references.citation: Am J Gastroenterol. 2007 Nov;102(11):2437-43
references.pmid: 17662100
references.citation: Gastroenterology. 2002 Jun;122(7):1822-8
references.pmid: 12055590
references.citation: CMAJ. 2011 Feb 22;183(3):E173-9
references.pmid: 21220436
references.citation: Epidemiology. 2011 Jul;22(4):450-6
references.pmid: 21490505
references.citation: JAMA. 2010 Aug 11;304(6):657-63
references.pmid: 20699457
references.citation: Aliment Pharmacol Ther. 2005 Aug 1;22(3):175-81
references.pmid: 16091054
references.citation: Aliment Pharmacol Ther. 2006 Jul 15;24(2):299-306
references.pmid: 16842456
references.citation: Gastrointest Endosc. 2000 Apr;51(4 Pt 1):423-6
references.pmid: 10744813