title: An empirical comparison of heterogeneity variance estimators in 12 894 meta-analyses.
abstract: Heterogeneity in meta-analysis is most commonly estimated using a moment-based approach described by DerSimonian and Laird. However, this method has been shown to produce biased estimates. Alternative methods to estimate heterogeneity include the restricted maximum likelihood approach and those proposed by Paule and Mandel, Sidik and Jonkman, and Hartung and Makambi. We compared the impact of these five methods on the results of 12,894 meta-analyses extracted from the Cochrane Database of Systematic Reviews. We compared the methods in terms of the following: (1) the extent of heterogeneity, expressed as an I(2) statistic; (2) the overall effect estimate; (3) the precision of the overall effect estimate; and (4) p-values testing the no effect hypothesis. Results suggest that, in some meta-analyses, I(2) estimates differ by more than 50% when different heterogeneity estimators are used. Conclusions naively based on statistical significance (at a 5% level) were discordant for at least one pair of estimators in 7.5% of meta-analyses, indicating that the choice of heterogeneity estimator could affect the conclusions of a meta-analysis. These findings imply that using a single estimate of heterogeneity may lead to non-robust results in some meta-analyses, and researchers should consider using alternatives to the DerSimonian and Laird method.
authors.affiliation: Centre for Reviews and Dissemination, University of York, York, YO10 5DD, UK.
authors.forename: Dean
authors.initials: D
authors.lastname: Langan
authors.affiliation: School of Social and Community Medicine, University of Bristol, Bristol, UK.
authors.forename: Julian P T
authors.initials: JP
authors.lastname: Higgins
authors.affiliation: Centre for Reviews and Dissemination, University of York, York, YO10 5DD, UK.
authors.forename: Mark
authors.initials: M
authors.lastname: Simmonds
mesh_terms: D000465:Algorithms; D000704:Analysis of Variance; D002986:Clinical Trials as Topic / Q000706:statistics & numerical data*; D000068598:Data Accuracy; D003627:Data Interpretation, Statistical; D019317:Evidence-Based Medicine / Q000592:standards; D015201:Meta-Analysis as Topic; D011786:Quality Control; D015203:Reproducibility of Results; D012107:Research Design; D012196:Review Literature as Topic; D012680:Sensitivity and Specificity
keywords: DerSimonian-Laird; heterogeneity; meta-analysis; random-effects