Evaluation of an automated procedure for scoring patient notes as part of a clinical skills examination.
OBJECTIVE
The purpose of the present study was to examine the extent to which an automated scoring procedure that emulates expert ratings with latent semantic analysis could be used to score the written patient note component of the proposed clinical skills examination (CSE).


METHODS
Human ratings for four CSE cases collected in 2002 were compared to automated holistic scores and to regression-based scores based on automated holistic and component scores.


CONCLUSIONS
Regression-based scores account for approximately half of the variance in the human ratings and are more highly correlated with the ratings than the scores produced from the automated algorithm. Implications of this study and suggestions for follow-up research are discussed.
Swygert
Kimberly
K
Measurement Consulting Services, NBME, Philadelphia, PA 19104, USA. kswygert@nbme.org
Margolis
Melissa
M
King
Ann
A
Siftar
Time
T
Clyman
Stephen
S
Hawkins
Richard
R
Clauser
Brian
B
D000465:Algorithms; D002983:Clinical Competence; D004521:Educational Measurement / Q000706:statistics & numerical data*; D006801:Humans; D008007:Licensure, Medical; D016014:Linear Models; D008487:Medical History Taking; D010808:Physical Examination; D012984:Software; D014481:United States