A comparison of alternative item weighting strategies on the data gathering component of a clinical skills performance assessment.
BACKGROUND
Checklist scores used to produce the data gathering score on the Step 2 CS examination are currently weighted using an algorithm based on expert judgment about the importance of the item. The present research was designed to compare this approach with alternative weighting strategies.


METHODS
Scores from 21,140 examinees who took the United States Medical Licensing Examination Step 2 between May 2006 and February 2007 were subjected to five weighting models: (1) a regression weights model, (2) a factor loading weights model, (3) a standardized response model, (4) an equal weights model, and (5) the operational expert-judgment weights model.


RESULTS
Alternative weighting procedures may have a significant impact on the reliability and validity of checklist scores.


CONCLUSIONS
The results suggest that the current weighting procedure is useful, and the regression-based model holds promise for practical application. The regression-based model produces scores that are more reliable than those produced by the current procedure and more strongly related to the external criteria.
Kahraman
Nilufer
N
nkahraman@nbme.org
Clauser
Brian E
BE
Margolis
Melissa J
MJ
D000465:Algorithms; D002983:Clinical Competence / Q000706:statistics & numerical data*; D015331:Cohort Studies; D005163:Factor Analysis, Statistical; D006801:Humans; D007600:Judgment; D008007:Licensure, Medical; D015233:Models, Statistical; D011594:Psychometrics; D015203:Reproducibility of Results; D012189:Retrospective Studies; D014481:United States