title: Using Automated Scoring to Evaluate Written Responses in English and French on a High-Stakes Clinical Competency Examination.
abstract: We present a framework for technology-enhanced scoring of bilingual clinical decision-making (CDM) questions using an open-source scoring technology and evaluate the strength of the proposed framework using operational data from the Medical Council of Canada Qualifying Examination. Candidates' responses from six write-in CDM questions were used to develop a three-stage-automated scoring framework. In Stage 1, the linguistic features from CDM responses were extracted. In Stage 2, supervised machine learning techniques were employed for developing the scoring models. In Stage 3, responses to six English and French CDM questions were scored using the scoring models from Stage 2. Of the 8,007 English and French CDM responses, 7,643 were accurately scored with an agreement rate of 95.4% between human and computer scoring. This result serves as an improvement of 5.4% when compared with the human inter-rater reliability. Our framework yielded scores similar to those of expert physician markers and could be used for clinical competency assessment.
authors.affiliation: Centre for Research in Applied Measurement and Evaluation, University of Alberta, Edmonton, Alberta, Canada syed.latifi@ualberta.ca.
authors.forename: Syed
authors.initials: S
authors.lastname: Latifi
authors.affiliation: Centre for Research in Applied Measurement and Evaluation, University of Alberta, Edmonton, Alberta, Canada.
authors.forename: Mark J
authors.initials: MJ
authors.lastname: Gierl
authors.affiliation: Medical Council of Canada, Ottawa, Ontario, Canada.
authors.forename: André-Philippe
authors.initials: AP
authors.lastname: Boulais
authors.affiliation: Medical Council of Canada, Ottawa, Ontario, Canada.
authors.forename: André F
authors.initials: AF
authors.lastname: De Champlain
mesh_terms: D002170:Canada; D002983:Clinical Competence; D000066491:Clinical Decision-Making; D004521:Educational Measurement / Q000379:methods* / Q000592:standards*; D001330:Electronic Data Processing / Q000592:standards*; D006801:Humans; D008007:Licensure, Medical; D015203:Reproducibility of Results; D014175:Translating
keywords: automated essay scoring; clinical competency assessment; clinical decision making; computerized assessment and evaluation; medical licensing examination