title: Approximate measurement invariance in cross-classified rater-mediated assessments.
abstract: An important assumption underlying meaningful comparisons of scores in rater-mediated assessments is that measurement is commensurate across raters. When raters differentially apply the standards established by an instrument, scores from different raters are on fundamentally different scales and no longer preserve a common meaning and basis for comparison. In this study, we developed a method to accommodate measurement noninvariance across raters when measurements are cross-classified within two distinct hierarchical units. We conceptualized random item effects cross-classified graded response models and used random discrimination and threshold effects to test, calibrate, and account for measurement noninvariance among raters. By leveraging empirical estimates of rater-specific deviations in the discrimination and threshold parameters, the proposed method allows us to identify noninvariant items and empirically estimate and directly adjust for this noninvariance within a cross-classified framework. Within the context of teaching evaluations, the results of a case study suggested substantial noninvariance across raters and that establishing an approximately invariant scale through random item effects improves model fit and predictive validity.
authors.affiliation: College of Education, Criminal Justice and Human Services, University of Cincinnati Cincinnati, OH, USA.
authors.forename: Ben
authors.initials: B
authors.lastname: Kelcey
authors.affiliation: Graduate School of Education, Harvard University Cambridge, MA, USA.
authors.forename: Dan
authors.initials: D
authors.lastname: McGinn
authors.affiliation: Graduate School of Education, Harvard University Cambridge, MA, USA.
authors.forename: Heather
authors.initials: H
authors.lastname: Hill
keywords: measurement equivalence; measurement invariance; multilevel item response models; random item effects; teaching