title: Examiner error in curriculum-based measurement of oral reading.
abstract: Although curriculum based measures of oral reading (CBM-R) have strong technical adequacy, there is still a reason to believe that student performance may be influenced by factors of the testing situation, such as errors examiners make in administering and scoring the test. This study examined the construct-irrelevant variance introduced by examiners using a cross-classified multilevel model. We sought to determine the extent of variance in student CBM-R scores attributable to examiners and, if present, the extent to which it was moderated by students' grade level and English learner (EL) status. Fit indices indicated that a cross-classified random effects model (CCREM) best fits the data with measures nested within students, students nested within schools, and examiners crossing schools. Intraclass correlations of the CCREM revealed that roughly 16% of the variance in student CBM-R scores was associated between examiners. The remaining variance was associated with the measurement level, 3.59%; between students, 75.23%; and between schools, 5.21%. Results were moderated by grade level but not by EL status. The discussion addresses the implications of this error for low-stakes and high-stakes decisions about students, teacher evaluation systems, and hypothesis testing in reading intervention research.
authors.affiliation: Center on Teaching and Learning, University of Oregon, USA. Electronic address: kellic@uoregon.edu.
authors.forename: Kelli D
authors.initials: KD
authors.lastname: Cummings
authors.affiliation: Educational Methodology, Policy, and Leadership, University of Oregon, USA.
authors.forename: Gina
authors.initials: G
authors.lastname: Biancarosa
authors.affiliation: Educational Methodology, Policy, and Leadership, University of Oregon, USA.
authors.forename: Andrew
authors.initials: A
authors.lastname: Schaper
authors.affiliation: Florida Center for Reading Research, Florida State University, USA.
authors.forename: Deborah K
authors.initials: DK
authors.lastname: Reed
mesh_terms: D002648:Child; D003479:Curriculum; D004521:Educational Measurement / Q000592:standards*; D005260:Female; D006801:Humans; D007807:Language Tests; D008297:Male; D011932:Reading
keywords: Assessment fidelity; CBM-R; Curriculum-based measurement; DIBELS; Progress monitoring; Reading assessment