title: Random effects structure for confirmatory hypothesis testing: Keep it maximal.
abstract: Linear mixed-effects models (LMEMs) have become increasingly prominent in psycholinguistics and related areas. However, many researchers do not seem to appreciate how random effects structures affect the generalizability of an analysis. Here, we argue that researchers using LMEMs for confirmatory hypothesis testing should minimally adhere to the standards that have been in place for many decades. Through theoretical arguments and Monte Carlo simulation, we show that LMEMs generalize best when they include the maximal random effects structurejustified by the design. The generalization performance of LMEMs includingdata-drivenrandom effects structures strongly depends upon modeling criteria and sample size, yielding reasonable results on moderately-sized samples when conservative criteria are used, but with little or no power advantage over maximal models. Finally, random-intercepts-only LMEMs used on within-subjects and/or within-items data from populations where subjects and/or items vary in their sensitivity to experimental manipulations always generalize worse than separateF1andF2tests, and in many cases, even worse thanF1alone. Maximal LMEMs should be the 'gold standard' for confirmatory hypothesis testing in psycholinguistics and beyond.
authors.affiliation: Institute of Neuroscience and Psychology, University of Glasgow, 58 Hillhead St., Glasgow G12 8QB, United Kingdom.
authors.forename: Dale J
authors.initials: DJ
authors.lastname: Barr
authors.affiliation: Department of Linguistics, University of California at San Diego, La Jolla, CA 92093-0108, USA.
authors.forename: Roger
authors.initials: R
authors.lastname: Levy
authors.affiliation: Institute of Neuroscience and Psychology, University of Glasgow, 58 Hillhead St., Glasgow G12 8QB, United Kingdom.
authors.forename: Christoph
authors.initials: C
authors.lastname: Scheepers
authors.affiliation: Department of Brain and Cognitive Sciences, Massachussetts Institute of Technology, Cambridge, MA 02139, USA.
authors.forename: Harry J
authors.initials: HJ
authors.lastname: Tily
keywords: Monte Carlo simulation; generalization; linear mixed-effects models; statistics