title: Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests.
abstract: Statistical inference in psychology has traditionally relied heavily on p-value significance testing. This approach to drawing conclusions from data, however, has been widely criticized, and two types of remedies have been advocated. The first proposal is to supplement p values with complementary measures of evidence, such as effect sizes. The second is to replace inference with Bayesian measures of evidence, such as the Bayes factor. The authors provide a practical comparison of p values, effect sizes, and default Bayes factors as measures of statistical evidence, using 855 recently published t tests in psychology. The comparison yields two main results. First, although p values and default Bayes factors almost always agree about what hypothesis is better supported by the data, the measures often disagree about the strength of this support; for 70% of the data sets for which the p value falls between .01 and .05, the default Bayes factor indicates that the evidence is only anecdotal. Second, effect sizes can provide additional evidence to p values and default Bayes factors. The authors conclude that the Bayesian approach is comparatively prudent, preventing researchers from overestimating the evidence in favor of an effect.
authors.affiliation: Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands wetzels.ruud@gmail.com.
authors.forename: Ruud
authors.initials: R
authors.lastname: Wetzels
authors.affiliation: Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
authors.forename: Dora
authors.initials: D
authors.lastname: Matzke
authors.affiliation: Department of Cognitive Sciences, University of California, Irvine.
authors.forename: Michael D
authors.initials: MD
authors.lastname: Lee
authors.affiliation: Department of Psychological Sciences, University of Missouri-Columbia.
authors.forename: Jeffrey N
authors.initials: JN
authors.lastname: Rouder
authors.affiliation: Department of Cognitive Sciences, University of California, Irvine.
authors.forename: Geoffrey J
authors.initials: GJ
authors.lastname: Iverson
authors.affiliation: Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
authors.forename: Eric-Jan
authors.initials: EJ
authors.lastname: Wagenmakers
keywords:      t test;     p value; Bayes factor; effect size; hypothesis testing