title: Evaluating the impact of database heterogeneity on observational study results.
abstract: Clinical studies that use observational databases to evaluate the effects of medical products have become commonplace. Such studies begin by selecting a particular database, a decision that published papers invariably report but do not discuss. Studies of the same issue in different databases, however, can and do generate different results, sometimes with strikingly different clinical implications. In this paper, we systematically study heterogeneity among databases, holding other study methods constant, by exploring relative risk estimates for 53 drug-outcome pairs and 2 widely used study designs (cohort studies and self-controlled case series) across 10 observational databases. When holding the study design constant, our analysis shows that estimated relative risks range from a statistically significant decreased risk to a statistically significant increased risk in 11 of 53 (21%) of drug-outcome pairs that use a cohort design and 19 of 53 (36%) of drug-outcome pairs that use a self-controlled case series design. This exceeds the proportion of pairs that were consistent across databases in both direction and statistical significance, which was 9 of 53 (17%) for cohort studies and 5 of 53 (9%) for self-controlled case series. Our findings show that clinical studies that use observational databases can be sensitive to the choice of database. More attention is needed to consider how the choice of data source may be affecting results.
authors: Department of Statistics, Columbia University, 1255 Amsterdam Avenue, New York, NY 10027, USA. david.madigan@columbia.edu
authors: David
authors: D
authors: Madigan
authors: Patrick B
authors: PB
authors: Ryan
authors: Martijn
authors: M
authors: Schuemie
authors: Paul E
authors: PE
authors: Stang
authors: J Marc
authors: JM
authors: Overhage
authors: Abraham G
authors: AG
authors: Hartzema
authors: Marc A
authors: MA
authors: Suchard
authors: William
authors: W
authors: DuMouchel
authors: Jesse A
authors: JA
authors: Berlin
mesh_terms: D015982:Bias; D015331:Cohort Studies; D018849:Controlled Clinical Trials as Topic; D003625:Data Collection; D016208:Databases, Factual / Q000706:statistics & numerical data*; D004341:Drug Evaluation / Q000379:methods* / Q000592:standards / Q000706:statistics & numerical data; D006801:Humans; D019370:Observation; D015203:Reproducibility of Results; D012107:Research Design; D012306:Risk; D016896:Treatment Outcome
keywords: database; heterogeneity; methods; population characteristics; reproducibility of results; surveillance
references: Stat Med. 2012 Dec 30;31(30):4401-15
references: 23015364
references: PLoS Med. 2005 Aug;2(8):e124
references: 16060722
references: Lancet. 1991 Nov 2;338(8775):1127-30
references: 1682553
references: Am J Epidemiol. 1994 Nov 1;140(9):783-7
references: 7977288
references: JAMA. 2000 Apr 19;283(15):2008-12
references: 10789670
references: PLoS One. 2008 Aug 28;3(8):e3081
references: 18769481
references: Pediatrics. 2010 May;125(5):921-30
references: 20403939
references: Basic Clin Pharmacol Toxicol. 2006 Mar;98(3):266-74
references: 16611201
references: BMJ. 2011 Nov 07;343:d7020
references: 22065658
references: Int J Epidemiol. 2001 Feb;30(1):1-11
references: 11171840
references: BMJ. 2003 Sep 6;327(7414):557-60
references: 12958120
references: Curr Drug Saf. 2011 Jul;6(3):132-3
references: 22122386
references: Aliment Pharmacol Ther. 2010 Apr;31(8):810-23
references: 20102352
references: Circulation. 2012 May 22;125(20):2520-44
references: 22514251
references: Hum Reprod Update. 2011 Mar-Apr;17(2):159-70
references: 20833638
references: Int J Epidemiol. 1988 Sep;17(3):680-5
references: 3272133
references: J Am Med Inform Assoc. 2012 Jan-Feb;19(1):54-60
references: 22037893
references: J Clin Epidemiol. 2005 Apr;58(4):323-37
references: 15862718
references: JAMA. 2010 Aug 11;304(6):657-63
references: 20699457
references: Epidemiol Rev. 1993;15(2):328-51
references: 8174661
references: Ann Intern Med. 2010 Nov 2;153(9):600-6
references: 21041580
references: Epidemiology. 2009 Jul;20(4):512-22
references: 19487948
references: BMJ. 2010 Sep 01;341:c4444
references: 20813820
references: Lancet. 2004 Dec 4-10;364(9450):2021-9
references: 15582059
references: Basic Clin Pharmacol Toxicol. 2006 Mar;98(3):311-3
references: 16611207