title: Type I error inflation in the traditional by-participant analysis to metamemory accuracy: a generalized mixed-effects model perspective.
abstract: In order to examine metacognitive accuracy (i.e., the relationship between metacognitive judgment and memory performance), researchers often rely on by-participant analysis, where metacognitive accuracy (e.g., resolution, as measured by the gamma coefficient or signal detection measures) is computed for each participant and the computed values are entered into group-level statistical tests such as the t test. In the current work, we argue that the by-participant analysis, regardless of the accuracy measurements used, would produce a substantial inflation of Type I error rates when a random item effect is present. A mixed-effects model is proposed as a way to effectively address the issue, and our simulation studies examining Type I error rates indeed showed superior performance of mixed-effects model analysis as compared to the conventional by-participant analysis. We also present real data applications to illustrate further strengths of mixed-effects model analysis. Our findings imply that caution is needed when using the by-participant analysis, and recommend the mixed-effects model analysis.
authors.affiliation: School of Psychology and Clinical Language Sciences, University of Reading.
authors.forename: Kou
authors.initials: K
authors.lastname: Murayama
authors.affiliation: School of Psychology and Clinical Language Sciences, University of Reading.
authors.forename: Michiko
authors.initials: M
authors.lastname: Sakaki
authors.affiliation: Department of Psychology, University of California, Los Angeles.
authors.forename: Veronica X
authors.initials: VX
authors.lastname: Yan
authors.affiliation: School of Systems Engineering, University of Reading.
authors.forename: Garry M
authors.initials: GM
authors.lastname: Smith
mesh_terms: D003071:Cognition / Q000502:physiology*; D003198:Computer Simulation; D006801:Humans; D007600:Judgment / Q000502:physiology*; D008568:Memory / Q000502:physiology*; D008960:Models, Psychological; D017603:Signal Detection, Psychological / Q000502:physiology*