title: The use of meta-analytic statistical significance testing.
abstract: Meta-analysis multiplicity, the concept of conducting multiple tests of statistical significance within one review, is an underdeveloped literature. We address this issue by considering how Type I errors can impact meta-analytic results, suggest how statistical power may be affected through the use of multiplicity corrections, and propose how meta-analysts should analyze multiple tests of statistical significance. The context for this study is a meta-review of meta-analyses published in two leading review journals in education and psychology. Our review of 130 meta-analyses revealed a strong reliance on statistical significance testing without consideration of Type I errors or the use of multiplicity corrections. In order to provide valid conclusions, meta-analysts must consider these issues prior to conducting the review.
authors.affiliation: Peabody Research Institute, Vanderbilt University, Nashville, TN, 37203, USA.
authors.forename: Joshua R
authors.initials: JR
authors.lastname: Polanin
authors.affiliation: School of Education, Loyola University Chicago, Chicago, IL, 60611, USA.
authors.forename: Terri D
authors.initials: TD
authors.lastname: Pigott
mesh_terms: D056808:Biostatistics / Q000379:methods*; D003627:Data Interpretation, Statistical; D004493:Education / Q000706:statistics & numerical data; D006801:Humans; D015201:Meta-Analysis as Topic; D015233:Models, Statistical; D011584:Psychology / Q000706:statistics & numerical data
keywords: Type I errors; meta‐analysis; meta‐review; moderator analyses; multiplicity corrections; power analysis; statistical significance testing