title: Association between analytic strategy and estimates of treatment outcomes in meta-analyses.
abstract: OBJECTIVE
  A persistent dilemma when performing meta-analyses is whether all available trials should be included in the meta-analysis.
  
  
  OBJECTIVE
  To compare treatment outcomes estimated by meta-analysis of all trials and several alternative analytic strategies: single most precise trial (ie, trial with the narrowest confidence interval), meta-analysis restricted to the 25% largest trials, limit meta-analysis (a meta-analysis model adjusted for small-study effect), and meta-analysis restricted to trials at low overall risk of bias.
  
  
  METHODS
  One hundred sixty-three meta-analyses published between 2008 and 2010 in high-impact-factor journals and between 2011 and 2013 in the Cochrane Database of Systematic Reviews: 92 (705 randomized clinical trials [RCTs]) with subjective outcomes and 71 (535 RCTs) with objective outcomes.
  
  
  RESULTS
  For each meta-analysis, the difference in treatment outcomes between meta-analysis of all trials and each alternative strategy, expressed as a ratio of odds ratios (ROR), was assessed considering the dependency between strategies. A difference greater than 30% was considered substantial. RORs were combined by random-effects meta-analysis models to obtain an average difference across the sample. An ROR greater than 1 indicates larger treatment outcomes with meta-analysis of all trials. Subjective and objective outcomes were analyzed separately.
  
  
  RESULTS
  Treatment outcomes were larger in the meta-analysis of all trials than in the single most precise trial (combined ROR, 1.13 [95% CI, 1.07-1.19]) for subjective outcomes and 1.03 (95% CI, 1.01-1.05) for objective outcomes). The difference in treatment outcomes between these strategies was substantial in 47 of 92 (51%) meta-analyses of subjective outcomes (meta-analysis of all trials showing larger outcomes in 40/47) and in 28 of 71 (39%) meta-analyses of objective outcomes (meta-analysis of all trials showing larger outcomes in 21/28). The combined ROR for subjective and objective outcomes was, respectively, 1.08 (95% CI, 1.04-1.13) and 1.03 (95% CI, 1.00-1.06) when comparing meta-analysis of all trials and meta-analysis of the 25% largest trials, 1.17 (95% CI, 1.11-1.22) and 1.13 (95% CI, 0.82-1.55) when comparing meta-analysis of all trials and limit meta-analysis, and 0.94 (95% CI, 0.86-1.04) and 1.03 (95% CI, 1.00-1.06) when comparing meta-analysis of all trials and meta-analysis restricted to trials at low risk of bias.
  
  
  CONCLUSIONS
  Estimation of treatment outcomes in meta-analyses differs depending on the strategy used. This instability in findings can result in major alterations in the conclusions derived from the analysis and underlines the need for systematic sensitivity analyses.
authors.affiliation: Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Paris, France2Centre d'Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique des Hôpitaux de Paris, Paris, France3Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cit.
authors.forename: Agnes
authors.initials: A
authors.lastname: Dechartres
authors.affiliation: Centre for Statistics in Medicine, Oxford, United Kingdom.
authors.forename: Douglas G
authors.initials: DG
authors.lastname: Altman
authors.affiliation: Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Paris, France5Department of Epidemiology, Mailman School of Public Health, Columbia University, New York, New York.
authors.forename: Ludovic
authors.initials: L
authors.lastname: Trinquart
authors.affiliation: Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Paris, France2Centre d'Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique des Hôpitaux de Paris, Paris, France3Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cit.
authors.forename: Isabelle
authors.initials: I
authors.lastname: Boutron
authors.affiliation: Centre de Recherche Epidémiologie et Statistique, INSERM U1153, Paris, France2Centre d'Épidémiologie Clinique, Hôpital Hôtel Dieu, Assistance Publique des Hôpitaux de Paris, Paris, France3Faculté de Médecine, Université Paris Descartes, Sorbonne Paris Cit.
authors.forename: Philippe
authors.initials: P
authors.lastname: Ravaud
mesh_terms: D016001:Confidence Intervals; D006801:Humans; D015201:Meta-Analysis as Topic; D016017:Odds Ratio; D016032:Randomized Controlled Trials as Topic; D012107:Research Design; D016896:Treatment Outcome