title: In the minds of OSCE examiners: uncovering hidden assumptions.
abstract: The Objective Structured Clinical Exam (OSCE) is a widely used method of assessment in medical education. Rater cognition has become an important area of inquiry in the medical education assessment literature generally, and in the OSCE literature specifically, because of concerns about potential compromises of validity. In this study, a novel approach to mixed methods that combined Ordinal Logistic Hierarchical Linear Modeling and cognitive interviews was used to gain insights about what examiners were thinking during an OSCE. This study is based on data from the 2010 to 2014 administrations of the Clinician Assessment for Practice Program OSCE for International Medical Graduates (IMGs) in Nova Scotia. An IMG is a physician trained outside of Canada who was a licensed practitioner in a different country. The quantitative data were examined alongside four follow-up cognitive interviews of examiners conducted after the 2014 administration. The quantitative results show that competencies of (1) Investigation and Management and (2) Counseling were highly predictive of the Overall Global score. These competencies were also described in the cognitive interviews as the most salient parts of OSCE. Examiners also found Communication Skills and Professional Behavior to be relevant but the quantitative results revealed these to be less predictive of the Overall Global score. The interviews also reveal that there is a tacit sequence by which IMGs are expected to proceed in an OSCE, starting with more basic competencies such as History Taking and building up to Investigation Management and Counseling. The combined results confirm that a hidden pattern exists with respect to how examiners rate candidates. This study has potential implications for research into rater cognition, and the design and scoring of practice-ready OSCEs.
authors.affiliation: Department of Medicine and Faculty of Education, Centre for Education Research and Innovation, Schulich School of Medicine and Dentistry, University of Western Ontario, Health Sciences Addition, Room 114, London, ON, N6A 5C1, Canada. saad.chahine@schulich.uwo.ca.
authors.forename: Saad
authors.initials: S
authors.lastname: Chahine
authors.affiliation: Division of Medical Education, Dalhousie University, Halifax, NS, Canada.
authors.forename: Bruce
authors.initials: B
authors.lastname: Holmes
authors.affiliation: Faculty of Education, Mount Saint Vincent University, Halifax, NS, Canada.
authors.forename: Zbigniew
authors.initials: Z
authors.lastname: Kowalewski
mesh_terms: D002983:Clinical Competence / Q000592:standards; D004501:Education, Medical / Q000592:standards*; D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D006801:Humans; D007407:Interviews as Topic; D016014:Linear Models; D016015:Logistic Models; D009674:Nova Scotia; D015203:Reproducibility of Results
keywords: Assessment; Cognitive interview; Hierarchical Linear Modeling (HLM); International Medical Graduate (IMG); Objective Structured Clinical Exam (OSCE); Practice-ready; Rater cognition