title: Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies.
abstract: Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.
authors.affiliation: Psychological Methods, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands. aoj.cramer@gmail.com.
authors.forename: Angélique O J
authors.initials: AO
authors.lastname: Cramer
authors.affiliation: Faculty of Science and Information Technology, School of Psychology, University of Newcastle, Callaghan, New South Wales, Australia.
authors.forename: Don
authors.initials: D
authors.lastname: van Ravenzwaaij
authors.affiliation: Psychological Methods, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
authors.forename: Dora
authors.initials: D
authors.lastname: Matzke
authors.affiliation: Psychological Methods, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
authors.forename: Helen
authors.initials: H
authors.lastname: Steingroever
authors.affiliation: Data Analytics, Price Waterhouse Coopers, Amsterdam, The Netherlands.
authors.forename: Ruud
authors.initials: R
authors.lastname: Wetzels
authors.affiliation: Psychological Methods, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
authors.forename: Raoul P P P
authors.initials: RP
authors.lastname: Grasman
authors.affiliation: Psychological Methods, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
authors.forename: Lourens J
authors.initials: LJ
authors.lastname: Waldorp
authors.affiliation: Psychological Methods, Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands.
authors.forename: Eric-Jan
authors.initials: EJ
authors.lastname: Wagenmakers
mesh_terms: D000704:Analysis of Variance; D035843:Biomedical Research / Q000592:standards*; D003627:Data Interpretation, Statistical; D006801:Humans; D011584:Psychology / Q000592:standards*
keywords: Benjamini–Hochberg procedure; Factorial ANOVA; False discovery rate; Familywise error rate; Multiple comparison problem; Multiway ANOVA; Preregistration; Sequential Bonferroni; Type I error