title: Exploring examiner judgement of professional competence in rater based assessment.
abstract: Exercise physiology courses have transitioned to competency based, forcing Universities to rethink assessment to ensure students are competent to practice. This study built on earlier research to explore rater cognition, capturing factors that contribute to assessor decision making about students' competency. The aims were to determine the source of variation in the examination process and document the factors impacting on examiner judgment. Examiner judgement was explored from both a quantitative and qualitative perspective. Twenty-three examiners viewed three video encounters of student performance on an OSCE. Once rated, analysis of variance was performed to determine where the variance was attributed. A semi-structured interview drew out the examiners reasoning behind their ratings. Results highlighted variability of the process of observation, judgement and rating, with each examiner viewing student performance from different lenses. However, at a global level, analysis of variance indicated that the examiner had a minimal impact on the variance, with the majority of variance explained by the student performance on task. One anomaly noted was in the assessment of technical competency, whereby the examiner had a large impact on the rating, linked to assessing according to curriculum content. The thought processes behind judgement were diverse and if the qualitative results had been used in isolation, may have led to the researchers drawing conclusions that the examined performances would have yielded widely different ratings. However, as a cohort, the examiners were able to distinguish good and poor levels of competency with the majority of student competency linked to the varying ability of the student.
authors.affiliation: Faculty of Health, School of Exercise and Nutrition Sciences, Queensland University of Technology, Musk Ave, Kelvin Grove, QLD, 4059, Australia. Fiona.naumann@qut.edu.au.
authors.forename: Fiona L
authors.initials: FL
authors.lastname: Naumann
authors.affiliation: UNSW Arts and Social Science, Learning and Teaching, The University of New South Wales Australia, Randwick, NSW, 2052, Australia.
authors.forename: Stephen
authors.initials: S
authors.lastname: Marshall
authors.affiliation: UNSW Medicine, Medical Education, The University of New South Wales Australia, Randwick, NSW, 2052, Australia.
authors.forename: Boaz
authors.initials: B
authors.lastname: Shulruf
authors.affiliation: UNSW Medicine, Medical Education, The University of New South Wales Australia, Randwick, NSW, 2052, Australia.
authors.forename: Philip D
authors.initials: PD
authors.lastname: Jones
mesh_terms: D001315:Australia; D002983:Clinical Competence / Q000592:standards*; D003162:Competency-Based Education / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D005260:Female; D006801:Humans; D007407:Interviews as Topic; D008297:Male; D010806:Physical Education and Training; D010827:Physiology / Q000193:education*; D014741:Video Recording
keywords: Assessment; Clinical competency; Examiner judgement; Exercise physiology