title: Is There a Free Lunch in Inference?
abstract: The field of psychology, including cognitive science, is vexed by a crisis of confidence. Although the causes and solutions are varied, we focus here on a common logical problem in inference. The default mode of inference is significance testing, which has a free lunch property where researchers need not make detailed assumptions about the alternative to test the null hypothesis. We present the argument that there is no free lunch; that is, valid testing requires that researchers test the null against a well-specified alternative. We show how this requirement follows from the basic tenets of conventional and Bayesian probability. Moreover, we show in both the conventional and Bayesian framework that not specifying the alternative may lead to rejections of the null hypothesis with scant evidence. We review both frequentist and Bayesian approaches to specifying alternatives, and we show how such specifications improve inference. The field of cognitive science will benefit because consideration of reasonable alternatives will undoubtedly sharpen the intellectual underpinnings of research.
authors.affiliation: University of Missouri.
authors.forename: Jeffrey N
authors.initials: JN
authors.lastname: Rouder
authors.affiliation: University of Cardiff.
authors.forename: Richard D
authors.initials: RD
authors.lastname: Morey
authors.affiliation: University of Amsterdam.
authors.forename: Josine
authors.initials: J
authors.lastname: Verhagen
authors.affiliation: University of Missouri.
authors.forename: Jordan M
authors.initials: JM
authors.lastname: Province
authors.affiliation: University of Amsterdam.
authors.forename: Eric-Jan
authors.initials: EJ
authors.lastname: Wagenmakers
mesh_terms: D001499:Bayes Theorem; D019336:Cognitive Science; D006801:Humans; D012107:Research Design
keywords: Inference; Philosophy of science; Replication crisis; Statistics