title: Small studies are more heterogeneous than large ones: a meta-meta-analysis.
abstract: OBJECTIVE
  Between-study heterogeneity plays an important role in random-effects models for meta-analysis. Most clinical trials are small, and small trials are often associated with larger effect sizes. We empirically evaluated whether there is also a relationship between trial size and heterogeneity (τ).
  
  
  METHODS
  We selected the first meta-analysis per intervention review of the Cochrane Database of Systematic Reviews Issues 2009-2013 with a dichotomous (n = 2,009) or continuous (n = 1,254) outcome. The association between estimated τ and trial size was evaluated across meta-analyses using regression and within meta-analyses using a Bayesian approach. Small trials were predefined as those having standard errors (SEs) over 0.2 standardized effects.
  
  
  RESULTS
  Most meta-analyses were based on few (median 4) trials. Within the same meta-analysis, the small study τS(2) was larger than the large-study τL(2) [average ratio 2.11; 95% credible interval (1.05, 3.87) for dichotomous and 3.11 (2.00, 4.78) for continuous meta-analyses]. The imprecision of τS was larger than of τL: median SE 0.39 vs. 0.20 for dichotomous and 0.22 vs. 0.13 for continuous small-study and large-study meta-analyses.
  
  
  CONCLUSIONS
  Heterogeneity between small studies is larger than between larger studies. The large imprecision with which τ is estimated in a typical small-studies' meta-analysis is another reason for concern, and sensitivity analyses are recommended.
authors.affiliation: Radboud Institute for Health Sciences (RIHS), Radboud University Medical Center, Mailbox 133, P.O. Box 9101, Nijmegen 6500 HB, The Netherlands. Electronic address: joanna.inthout@radboudumc.nl.
authors.forename: Joanna
authors.initials: J
authors.lastname: IntHout
authors.affiliation: Stanford Prevention Research Center, Department of Medicine, Stanford University School of Humanities and Sciences, 1265 Welch Road, Stanford, CA 94305, USA; Department of Health Research and Policy, Stanford University School of Medicine, 150 Governor's Lane, Stanford, CA 94305, USA; Department of Statistics, Stanford University School of Humanities and Sciences, 390 Serra Mall, Stanford, CA 94305, USA; Meta-Research Innovation Center at Stanford (METRICS), Stanford University, 1265 Welch Road, Stanford, CA 94305, USA.
authors.forename: John P A
authors.initials: JP
authors.lastname: Ioannidis
authors.affiliation: Radboud Institute for Health Sciences (RIHS), Radboud University Medical Center, Mailbox 133, P.O. Box 9101, Nijmegen 6500 HB, The Netherlands.
authors.forename: George F
authors.initials: GF
authors.lastname: Borm
authors.affiliation: Radboud Institute for Health Sciences (RIHS), Radboud University Medical Center, Mailbox 133, P.O. Box 9101, Nijmegen 6500 HB, The Netherlands.
authors.forename: Jelle J
authors.initials: JJ
authors.lastname: Goeman
mesh_terms: D001499:Bayes Theorem; D002986:Clinical Trials as Topic; D004812:Epidemiologic Methods; D006801:Humans; D008962:Models, Theoretical; D012107:Research Design
keywords: Between-study heterogeneity; Cochrane Database of systematic reviews (CDSR); Meta-analysis; Random-effects model; Randomized controlled trial; Trial size