title: Synthesizing Evidence in Public Policy Contexts: The Challenge of Synthesis When There Are Only a Few Studies.
abstract: For a variety of reasons, researchers and evidence-based clearinghouses synthesizing the results of multiple studies often have very few studies that are eligible for any given research question. This situation is less than optimal for meta-analysis as it is usually practiced, that is, by employing inverse variance weights, which allows more informative studies to contribute relatively more to the analysis. This article outlines the choices available for synthesis when there are few studies to synthesize. As background, we review the synthesis practices used in several projects done at the behest of governmental agencies and private foundations. We then discuss the strengths and limitations of different approaches to meta-analysis in a limited information environment. Using examples from the U.S. Department of Education's What Works Clearinghouse as case studies, we conclude with a discussion of Bayesian meta-analysis as a potential solution to the challenges encountered when attempting to draw inferences about the effectiveness of interventions from a small number of studies.
authors: 1 University of Louisville, Louisville, KY, USA.
authors: Jeffrey C
authors: JC
authors: Valentine
authors: 2 Vanderbilt University, Nashville, USA.
authors: Sandra Jo
authors: SJ
authors: Wilson
authors: 3 The Graduate Center, City University of New York, New York, NY, USA.
authors: David
authors: D
authors: Rindskopf
authors: 1 University of Louisville, Louisville, KY, USA.
authors: Timothy S
authors: TS
authors: Lau
authors: 2 Vanderbilt University, Nashville, USA.
authors: Emily E
authors: EE
authors: Tanner-Smith
authors: 4 Development Services Group, Bethesda, MD, USA.
authors: Martha
authors: M
authors: Yeide
authors: 4 Development Services Group, Bethesda, MD, USA.
authors: Robin
authors: R
authors: LaSota
authors: 4 Development Services Group, Bethesda, MD, USA.
authors: Lisa
authors: L
authors: Foster
keywords: Bayesian statistics; content area; education; meta-analysis; methodological development; research synthesis; systematic review