title: Inter-rater variability as mutual disagreement: identifying raters' divergent points of view.
abstract: Whenever multiple observers provide ratings, even of the same performance, inter-rater variation is prevalent. The resulting 'idiosyncratic rater variance' is considered to be unusable error of measurement in psychometric models and is a threat to the defensibility of our assessments. Prior studies of inter-rater variation in clinical assessments have used open response formats to gather raters' comments and justifications. This design choice allows participants to use idiosyncratic response styles that could result in a distorted representation of the underlying rater cognition and skew subsequent analyses. In this study we explored rater variability using the structured response format of Q methodology. Physician raters viewed video-recorded clinical performances and provided Mini Clinical Evaluation Exercise (Mini-CEX) assessment ratings through a web-based system. They then shared their assessment impressions by sorting statements that described the most salient aspects of the clinical performance onto a forced quasi-normal distribution ranging from "most consistent with my impression" to "most contrary to my impression". Analysis of the resulting Q-sorts revealed distinct points of view for each performance shared by multiple physicians. The points of view corresponded with the ratings physicians assigned to the performance. Each point of view emphasized different aspects of the performance with either rapport-building and/or medical expertise skills being most salient. It was rare for the points of view to diverge based on disagreements regarding the interpretation of a specific aspect of the performance. As a result, physicians' divergent points of view on a given clinical performance cannot be easily reconciled into a single coherent assessment judgment that is impacted by measurement error. If inter-rater variability does not wholly reflect error of measurement, it is problematic for our current measurement models and poses challenges for how we are to adequately analyze performance assessment ratings.
authors.affiliation: Northern Medical Program, University of Northern British Columbia, 3333 University Way, Prince George, BC, V2N 4Z9, Canada. andrea.gingerich@unbc.ca.
authors.forename: Andrea
authors.identifier: http://orcid.org/0000-0001-5765-3975
authors.initials: A
authors.lastname: Gingerich
authors.affiliation: Department of Engineering and Science Technology, University of Akron, Akron, OH, USA.
authors.forename: Susan E
authors.initials: SE
authors.lastname: Ramlo
authors.affiliation: School of Health Professions Education, Maastricht University, Maastricht, Netherlands.
authors.forename: Cees P M
authors.initials: CPM
authors.lastname: van der Vleuten
authors.affiliation: Centre for Health Education Scholarship, University of British Columbia, Vancouver, BC, Canada.
authors.forename: Kevin W
authors.initials: KW
authors.lastname: Eva
authors.affiliation: Centre for Health Education Scholarship, University of British Columbia, Vancouver, BC, Canada.
authors.forename: Glenn
authors.initials: G
authors.lastname: Regehr
mesh_terms: D002983:Clinical Competence; D004501:Education, Medical / Q000379:methods* / Q000592:standards*; D004521:Educational Measurement / Q000379:methods* / Q000592:standards*; D005260:Female; D006801:Humans; D008297:Male; D015588:Observer Variation; D015203:Reproducibility of Results; D014741:Video Recording
keywords: Inter-rater variability; Mini-CEX; Q methodology; Rater cognition; Rater-based assessment; Workplace-based assessment