title: Does design matter? Systematic evaluation of the impact of analytical choices on effect estimates in observational studies.
abstract: BACKGROUND
  Clinical studies that use observational databases, such as administrative claims and electronic health records, to evaluate the effects of medical products have become commonplace. These studies begin by selecting a particular study design, such as a case control, cohort, or self-controlled design, and different authors can and do choose different designs for the same clinical question. Furthermore, published papers invariably report the study design but do not discuss the rationale for the specific choice. Studies of the same clinical question with different designs, however, can generate different results, sometimes with strikingly different implications. Even within a specific study design, authors make many different analytic choices and these too can profoundly impact results. In this paper, we systematically study heterogeneity due to the type of study design and due to analytic choices within study design.
  
  
  RESULTS
  We conducted our analysis in 10 observational healthcare databases but mostly present our results in the context of the GE Centricity EMR database, an electronic health record database containing data for 11.2 million lives. We considered the impact of three different study design choices on estimates of associations between bisphosphonates and four particular health outcomes for which there is no evidence of an association. We show that applying alternative study designs can yield discrepant results, in terms of direction and significance of association. We also highlight that while traditional univariate sensitivity analysis may not show substantial variation, systematic assessment of all analytical choices within a study design can yield inconsistent results ranging from statistically significant decreased risk to statistically significant increased risk. Our findings show that clinical studies using observational databases can be sensitive both to study design choices and to specific analytic choices within study design.
  
  
  CONCLUSIONS
  More attention is needed to consider how design choices may be impacting results and, when possible, investigators should examine a wide array of possible choices to confirm that significant findings are consistently identified.
authors.affiliation: Professor and Chair, Department of Statistics, Columbia University, 1255 Amsterdam Ave., New York, NY 10027, USA.
authors.forename: David
authors.initials: D
authors.lastname: Madigan
authors.affiliation: Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD and Janssen Research and Development LLC, Titusville, NJ, USA.
authors.forename: Patrick B
authors.initials: PB
authors.lastname: Ryan
authors.affiliation: Observational Medical Outcomes Partnership, Foundation for the National Institutes of Health, Bethesda, MD and Department of Medical Informatics, Erasmus University Medical Center Rotterdam, Rotterdam, The Netherlands.
authors.forename: Martijn
authors.initials: M
authors.lastname: Schuemie
keywords: analysis; health outcomes; healthcare database; study design
references.citation: JAMA. 2001 Apr 11;285(14):1850-5
references.pmid: 11308398
references.citation: Stat Med. 2012 Dec 30;31(30):4401-15
references.pmid: 23015364
references.citation: PLoS Med. 2005 Aug;2(8):e124
references.pmid: 16060722
references.citation: BMJ. 2008 Aug 28;337:a1069
references.pmid: 18755768
references.citation: Pharmacoepidemiol Drug Saf. 2012 Jan;21 Suppl 1:32-40
references.pmid: 22262591
references.citation: Popul Health Manag. 2010 Jun;13(3):139-50
references.pmid: 20568974
references.citation: Int J Epidemiol. 2006 Oct;35(5):1301-8
references.pmid: 17053011
references.citation: JAMA. 2010 Aug 11;304(6):657-63
references.pmid: 20699457
references.citation: Pharmacoepidemiol Drug Saf. 2007 Mar;16(3):329-36
references.pmid: 16741894
references.citation: Ann Intern Med. 2010 Nov 2;153(9):600-6
references.pmid: 21041580
references.citation: J Biomed Inform. 2012 Aug;45(4):689-96
references.pmid: 22683994
references.citation: Am J Med Qual. 2008 Sep-Oct;23(5):375-81
references.pmid: 18820142
references.citation: PLoS One. 2009;4(3):e4720
references.pmid: 19266096
references.citation: Int J Epidemiol. 1988 Sep;17(3):680-5
references.pmid: 3272133
references.citation: J Am Med Inform Assoc. 2012 Jan-Feb;19(1):54-60
references.pmid: 22037893
references.citation: JAMA. 2000 Jun 28;283(24):3205-10
references.pmid: 10866867
references.citation: J Clin Epidemiol. 2008 Apr;61(4):344-9
references.pmid: 18313558
references.citation: BMJ. 2011 Mar 17;342:d1309
references.pmid: 21415101
references.citation: Epidemiology. 2009 Jul;20(4):512-22
references.pmid: 19487948
references.citation: BMJ. 2010 Sep 01;341:c4444
references.pmid: 20813820
references.citation: Nat Rev Rheumatol. 2011 Jun;7(6):369-72
references.pmid: 21537362
references.citation: Basic Clin Pharmacol Toxicol. 2006 Mar;98(3):266-74
references.pmid: 16611201
references.citation: Am J Epidemiol. 2004 Aug 15;160(4):384-92
references.pmid: 15286024
references.citation: Pharmacoepidemiol Drug Saf. 2009 Dec;18(12):1176-84
references.pmid: 19757412
references.citation: Int J Epidemiol. 2001 Feb;30(1):1-11
references.pmid: 11171840