title: Comparison of evidence of treatment effects in randomized and nonrandomized studies.
abstract: BACKGROUND
  There is substantial debate about whether the results of nonrandomized studies are consistent with the results of randomized controlled trials on the same topic.
  
  
  OBJECTIVE
  To compare results of randomized and nonrandomized studies that evaluated medical interventions and to examine characteristics that may explain discrepancies between randomized and nonrandomized studies.
  
  
  METHODS
  MEDLINE (1966-March 2000), the Cochrane Library (Issue 3, 2000), and major journals were searched.
  
  
  METHODS
  Forty-five diverse topics were identified for which both randomized trials (n = 240) and nonrandomized studies (n = 168) had been performed and had been considered in meta-analyses of binary outcomes.
  
  
  METHODS
  Data on events per patient in each study arm and design and characteristics of each study considered in each meta-analysis were extracted and synthesized separately for randomized and nonrandomized studies.
  
  
  RESULTS
  Very good correlation was observed between the summary odds ratios of randomized and nonrandomized studies (r = 0.75; P<.001); however, nonrandomized studies tended to show larger treatment effects (28 vs 11; P =.009). Between-study heterogeneity was frequent among randomized trials alone (23%) and very frequent among nonrandomized studies alone (41%). The summary results of the 2 types of designs differed beyond chance in 7 cases (16%). Discrepancies beyond chance were less common when only prospective studies were considered (8%). Occasional differences in sample size and timing of publication were also noted between discrepant randomized and nonrandomized studies. In 28 cases (62%), the natural logarithm of the odds ratio differed by at least 50%, and in 15 cases (33%), the odds ratio varied at least 2-fold between nonrandomized studies and randomized trials.
  
  
  CONCLUSIONS
  Despite good correlation between randomized trials and nonrandomized studies-in particular, prospective studies-discrepancies beyond chance do occur and differences in estimated magnitude of treatment effect are very common.
authors.affiliation: Division of Clinical Care Research, New England Medical Center, Box 63, 750 Washington St, Boston, MA 02111, USA. JLau1@lifespan.org
authors.forename: J P
authors.initials: JP
authors.lastname: Ioannidis
authors.forename: A B
authors.initials: AB
authors.lastname: Haidich
authors.forename: M
authors.initials: M
authors.lastname: Pappa
authors.forename: N
authors.initials: N
authors.lastname: Pantazis
authors.forename: S I
authors.initials: SI
authors.lastname: Kokori
authors.forename: M G
authors.initials: MG
authors.lastname: Tektonidou
authors.forename: D G
authors.initials: DG
authors.lastname: Contopoulos-Ioannidis
authors.forename: J
authors.initials: J
authors.lastname: Lau
mesh_terms: D002986:Clinical Trials as Topic; D003627:Data Interpretation, Statistical; D019317:Evidence-Based Medicine; D006801:Humans; D015201:Meta-Analysis as Topic; D016032:Randomized Controlled Trials as Topic