Scoring the computer-based case simulation component of USMLE Step 3: a comparison of preoperational and operational data.
OBJECTIVE
Operational USMLE(TM) computer-based case simulation results were examined to determine the extent to which rater reliability and regression model performance met expectations based on preoperational data.


METHODS
Operational data resulted from Step 3 examinations given between 1999 and 2004. Plots were produced using reliability and multiple correlation coefficients.


RESULTS
Operational testing reliabilities increased over the four years but were lower than the preoperational reliability. Multiple correlation coefficient results are somewhat superior to the results reported during the preoperational period and suggest that the operational scoring algorithms have been relatively consistent.


CONCLUSIONS
Changes in the rater population, changes in the rating task, and enhancements to the training procedures are several factors that can explain the identified differences between preoperational and operational results. The present findings have important implications for test development and test validity.
Margolis
Melissa J
MJ
National Board of Medical Examiners, Philadelphia, PA 19104, USA. mmargolis@nbme.org
Clauser
Brian E
BE
Harik
Polina
P
D000465:Algorithms; D002983:Clinical Competence; D003198:Computer Simulation; D004501:Education, Medical; D004521:Educational Measurement / Q000379:methods* / Q000706:statistics & numerical data; D006801:Humans; D008007:Licensure, Medical; D015588:Observer Variation; D012044:Regression Analysis; D015203:Reproducibility of Results