title: Agreement of treatment effects for mortality from routinely collected data and subsequent randomized trials: meta-epidemiological survey.
abstract: OBJECTIVE
  To assess differences in estimated treatment effects for mortality between observational studies with routinely collected health data (RCD; that are published before trials are available) and subsequent evidence from randomized controlled trials on the same clinical question.
  
  
  METHODS
  Meta-epidemiological survey.
  
  
  METHODS
  PubMed searched up to November 2014.
  
  
  METHODS
  Eligible RCD studies were published up to 2010 that used propensity scores to address confounding bias and reported comparative effects of interventions for mortality. The analysis included only RCD studies conducted before any trial was published on the same topic. The direction of treatment effects, confidence intervals, and effect sizes (odds ratios) were compared between RCD studies and randomized controlled trials. The relative odds ratio (that is, the summary odds ratio of trial(s) divided by the RCD study estimate) and the summary relative odds ratio were calculated across all pairs of RCD studies and trials. A summary relative odds ratio greater than one indicates that RCD studies gave more favorable mortality results.
  
  
  RESULTS
  The evaluation included 16 eligible RCD studies, and 36 subsequent published randomized controlled trials investigating the same clinical questions (with 17,275 patients and 835 deaths). Trials were published a median of three years after the corresponding RCD study. For five (31%) of the 16 clinical questions, the direction of treatment effects differed between RCD studies and trials. Confidence intervals in nine (56%) RCD studies did not include the RCT effect estimate. Overall, RCD studies showed significantly more favorable mortality estimates by 31% than subsequent trials (summary relative odds ratio 1.31 (95% confidence interval 1.03 to 1.65; I(2)=0%)).
  
  
  CONCLUSIONS
  Studies of routinely collected health data could give different answers from subsequent randomized controlled trials on the same clinical questions, and may substantially overestimate treatment effects. Caution is needed to prevent misguided clinical decision making.
authors: Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305, USA Basel Institute for Clinical Epidemiology and Biostatistics, University Hospital Basel, Basel, Switzerland.
authors: Lars G
authors: LG
authors: Hemkens
authors: Department of Pediatrics, Division of Infectious Diseases, Stanford University School of Medicine, Stanford, California, USA Meta-Research Innovation Center at Stanford (METRICS).
authors: Despina G
authors: DG
authors: Contopoulos-Ioannidis
authors: Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Stanford, CA 94305, USA Meta-Research Innovation Center at Stanford (METRICS) Department of Health Research and Policy, Stanford University School of Medicine, Stanford, California, USA Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, California, USA jioannid@stanford.edu.
authors: John P A
authors: JP
authors: Ioannidis
mesh_terms: D006801:Humans; D015201:Meta-Analysis as Topic; D064887:Observational Studies as Topic; D017063:Outcome Assessment, Health Care / Q000379:methods*; D057216:Propensity Score; D016032:Randomized Controlled Trials as Topic