title: Systematic Differences between Cochrane and Non-Cochrane Meta-Analyses on the Same Topic: A Matched Pair Analysis.
abstract: BACKGROUND
  Meta-analyses conducted via the Cochrane Collaboration adhere to strict methodological and reporting standards aiming to minimize bias, maximize transparency/reproducibility, and improve the accuracy of summarized data. Whether this results in differences in the results reported by meta-analyses on the same topic conducted outside the Cochrane Collaboration is an open question.
  
  
  METHODS
  We conducted a matched-pair analysis with individual meta-analyses as the unit of analysis, comparing Cochrane and non-Cochrane reviews. Using meta-analyses from the cardiovascular literature, we identified pairs that matched on intervention and outcome. The pairs were contrasted in terms of how frequently results disagreed between the Cochrane and non-Cochrane reviews, whether effect sizes and statistical precision differed systematically, and how these differences related to the frequency of secondary citations of those reviews.
  
  
  RESULTS
  Our search yielded 40 matched pairs of reviews. The two sets were similar in terms of which was first to publication, how many studies were included, and average sample sizes. The paired reviews included a total of 344 individual clinical trials: 111 (32.3%) studies were included only in a Cochrane review, 104 (30.2%) only in a non-Cochrane review, and 129 (37.5%) in both. Stated another way, 62.5% of studies were only included in one or the other meta-analytic literature. Overall, 37.5% of pairs had discrepant results. The most common involved shifts in the width of 95% confidence intervals that would yield a different statistical interpretation of the significance of results (7 pairs). Additionally, 20% differed in the direction of the summary effect size (5 pairs) or reported greater than a 2-fold difference in its magnitude (3 pairs). Non-Cochrane reviews reported significantly higher effect sizes (P< 0.001) and lower precision (P<0.001) than matched Cochrane reviews. Reviews reporting an effect size at least 2-fold greater than their matched pair were cited more frequently.
  
  
  CONCLUSIONS
  Though results between topic-matched Cochrane and non-Cochrane reviews were quite similar, discrepant results were frequent, and the overlap of included studies was surprisingly low. Non-Cochrane reviews report larger effect sizes with lower precision than Cochrane reviews, indicating systematic differences, likely reflective of methodology, between the two types of reviews that could generate different interpretations of the interventions under question.
authors.affiliation: Department of Epidemiology, Boston University School of Public Health, Boston, Massachusetts, United States of America.
authors.forename: Johanna
authors.initials: J
authors.lastname: Useem
authors.affiliation: Department of Epidemiology, Boston University School of Public Health, Boston, Massachusetts, United States of America.|Center for Global Health & Development, Boston University School of Public Health, Boston, Massachusetts, United States of America.|Health Economics and Epidemiology Research Office, Department of Internal Medicine, School of Clinical Medicine, Faculty of Health Sciences, University of the Witwatersrand, Johannesburg, South Africa.
authors.forename: Alana
authors.initials: A
authors.lastname: Brennan
authors.affiliation: Department of Biostatistics, Boston University School of Public Health, Boston, Massachusetts, United States of America.
authors.forename: Michael
authors.initials: M
authors.lastname: LaValley
authors.affiliation: Department of Global Health, Boston University School of Public Health, Boston, Massachusetts, United States of America.
authors.forename: Michelle
authors.initials: M
authors.lastname: Vickery
authors.affiliation: Department of Global Health, Boston University School of Public Health, Boston, Massachusetts, United States of America.
authors.forename: Omid
authors.initials: O
authors.lastname: Ameli
authors.affiliation: Department of Global Health, Boston University School of Public Health, Boston, Massachusetts, United States of America.
authors.forename: Nichole
authors.initials: N
authors.lastname: Reinen
authors.affiliation: Center for Global Health & Development, Boston University School of Public Health, Boston, Massachusetts, United States of America.|Department of Global Health, Boston University School of Public Health, Boston, Massachusetts, United States of America.
authors.forename: Christopher J
authors.initials: CJ
authors.lastname: Gill
mesh_terms: D006801:Humans; D016555:Matched-Pair Analysis; D011643:Publishing