title: Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects.
abstract: OBJECTIVE
  To assess whether nominally statistically significant effects in meta-analyses of clinical trials are true and whether their magnitude is inflated.
  
  
  METHODS
  Data from the Cochrane Database of Systematic Reviews 2005 (issue 4) and 2010 (issue 1) were used. We considered meta-analyses with binary outcomes and four or more trials in 2005 with P<0.05 for the random-effects odds ratio (OR). We examined whether any of these meta-analyses had updated counterparts in 2010. We estimated the credibility (true-positive probability) under different prior assumptions and inflation in OR estimates in 2005.
  
  
  RESULTS
  Four hundred sixty-one meta-analyses in 2005 were eligible, and 80 had additional trials included by 2010. The effect sizes (ORs) were smaller in the updating data (2005-2010) than in the respective meta-analyses in 2005 (median 0.85-fold, interquartile range [IQR]: 0.66-1.06), even more prominently for meta-analyses with less than 300 events in 2005 (median 0.67-fold, IQR: 0.54-0.96). Mean credibility of the 461 meta-analyses in 2005 was 63-84% depending on the assumptions made. Credibility estimates changed >20% in 19-31 (24-39%) of the 80 updated meta-analyses.
  
  
  CONCLUSIONS
  Most meta-analyses with nominally significant results pertain to truly nonnull effects, but exceptions are not uncommon. The magnitude of observed effects, especially in meta-analyses with limited evidence, is often inflated.
authors: Department of Hygiene and Epidemiology, Clinical Trials and Evidence-Based Medicine Unit, University of Ioannina School of Medicine, Ioannina 45110, Greece.
authors: Tiago V
authors: TV
authors: Pereira
authors: John P A
authors: JP
authors: Ioannidis
mesh_terms: D015982:Bias; D002986:Clinical Trials as Topic / Q000592:standards*; D006302:Health Services Research / Q000379:methods / Q000592:standards; D006801:Humans; D015201:Meta-Analysis as Topic