Weighting checklist items and station components on a large-scale OSCE: is it worth the effort?
BACKGROUND
Past research suggests that the use of externally-applied scoring weights may not appreciably impact measurement qualities such as reliability or validity. Nonetheless, some credentialing boards and academic institutions apply differential scoring weights based on expert opinion about the relative importance of individual items or test components of Observed Structured Clinical Examinations (OSCEs).


OBJECTIVE
To investigate the impact of simplified scoring models that make little to no use of differential weighting on the reliability of scores and decisions on a high stakes OSCE required for medical licensure in Canada.


METHODS
We applied four different weighting models of various complexities to data from three administrations of the OSCE. We compared score reliability, pass/fail rates, correlations between the scores and classification decision accuracy and consistency across the models and administrations.


RESULTS
Less complex weighting models yielded similar reliability and pass rates as the more complex weighting model. Minimal changes in candidates' pass/fail status were observed and there were strong and statistically significant correlations between the scores for all scoring models and administrations. Classification decision accuracy and consistency were very high and similar across the four scoring models.


CONCLUSIONS
Adopting a simplified weighting scheme for this OSCE did not diminish its measurement qualities. Instead of developing complex weighting schemes, experts' time and effort could be better spent on other critical test development and assembly tasks with little to no compromise in the quality of scores and decisions on this high-stakes OSCE.
University of British Columbia , Canada .
Debra Dallie
DD
Sandilands
Andrea
A
Gotzmann
Marguerite
M
Roy
Bruno D
BD
Zumbo
André
A
De Champlain
D002170:Canada; D057189:Checklist; D002983:Clinical Competence / Q000592:standards*; D004521:Educational Measurement / Q000379:methods / Q000592:standards* / Q000706:statistics & numerical data; D006801:Humans; D008007:Licensure, Medical / Q000592:standards*; D017145:Models, Educational; D015203:Reproducibility of Results