title: Meta-analyses are no substitute for registered replications: a skeptical perspective on religious priming.
abstract: According to a recent meta-analysis, religious priming has a positive effect on prosocial behavior (Shariff et al., 2015). We first argue that this meta-analysis suffers from a number of methodological shortcomings that limit the conclusions that can be drawn about the potential benefits of religious priming. Next we present a re-analysis of the religious priming data using two different meta-analytic techniques. A Precision-Effect Testing-Precision-Effect-Estimate with Standard Error (PET-PEESE) meta-analysis suggests that the effect of religious priming is driven solely by publication bias. In contrast, an analysis using Bayesian bias correction suggests the presence of a religious priming effect, even after controlling for publication bias. These contradictory statistical results demonstrate that meta-analytic techniques alone may not be sufficiently robust to firmly establish the presence or absence of an effect. We argue that a conclusive resolution of the debate about the effect of religious priming on prosocial behavior - and about theoretically disputed effects more generally - requires a large-scale, preregistered replication project, which we consider to be the sole remedy for the adverse effects of experimenter bias and publication bias.
authors.affiliation: Department of Psychology, University of Amsterdam Amsterdam, Netherlands.
authors.forename: Michiel
authors.initials: M
authors.lastname: van Elk
authors.affiliation: Department of Psychology, University of Amsterdam Amsterdam, Netherlands.
authors.forename: Dora
authors.initials: D
authors.lastname: Matzke
authors.affiliation: Department of Psychology, University of Amsterdam Amsterdam, Netherlands.
authors.forename: Quentin F
authors.initials: QF
authors.lastname: Gronau
authors.affiliation: Cognition and Individual Differences Lab, Department of Cognitive Sciences, University of California at Irvine Irvine, CA, USA.
authors.forename: Maime
authors.initials: M
authors.lastname: Guan
authors.affiliation: Cognition and Individual Differences Lab, Department of Cognitive Sciences, University of California at Irvine Irvine, CA, USA.
authors.forename: Joachim
authors.initials: J
authors.lastname: Vandekerckhove
authors.affiliation: Department of Psychology, University of Amsterdam Amsterdam, Netherlands.
authors.forename: Eric-Jan
authors.initials: EJ
authors.lastname: Wagenmakers
keywords: Bayesian Bias Correction; PET-PEESE; meta-analysis; preregistration; prosocial behavior; religious priming