title: Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011).
abstract: Does psi exist? D. J. Bem (2011) conducted 9 studies with over 1,000 participants in an attempt to demonstrate that future events retroactively affect people's responses. Here we discuss several limitations of Bem's experiments on psi; in particular, we show that the data analysis was partly exploratory and that one-sided p values may overstate the statistical evidence against the null hypothesis. We reanalyze Bem's data with a default Bayesian t test and show that the evidence for psi is weak to nonexistent. We argue that in order to convince a skeptical audience of a controversial claim, one needs to conduct strictly confirmatory studies and analyze the results with statistical tests that are conservative rather than liberal. We conclude that Bem's p values do not indicate evidence in favor of precognition; instead, they indicate that experimental psychologists need to change the way they conduct their experiments and analyze their data.
authors.affiliation: Department of Psychology, University of Amsterdam. ej.wagenmakers@gmail.com
authors.forename: Eric-Jan
authors.initials: EJ
authors.lastname: Wagenmakers
authors.forename: Ruud
authors.initials: R
authors.lastname: Wetzels
authors.forename: Denny
authors.initials: D
authors.lastname: Borsboom
authors.forename: Han L J
authors.initials: HL
authors.lastname: van der Maas
mesh_terms: D001499:Bayes Theorem; D035841:Behavioral Research / Q000592:standards / Q000706:statistics & numerical data; D003071:Cognition; D003627:Data Interpretation, Statistical; D017408:Guidelines as Topic / Q000592:standards; D006801:Humans; D010268:Parapsychology; D011584:Psychology / Q000379:methods* / Q000706:statistics & numerical data