Assessing the impact of modifications to the documentation component's scoring rubric and rater training on USMLE integrated clinical encounter scores.
BACKGROUND
Documentation is a subcomponent of the Step 2 Clinical Skills Examination Integrated Clinical Encounter (ICE) component wherein licensed physicians rate examinees on their ability to communicate the findings of the patient encounter, diagnostic impression, and initial patient work-up. The main purpose of this research was to examine the impact of modifications to the scoring rubric and rater training protocol on the psychometric characteristics of the documentation scores.


METHODS
Following the modifications, the variance structure of the ICE components was modeled using multivariate generalizability theory.


RESULTS
The results confirmed the expectations that true score variance for the documentation subcomponent would increase after adopting a modified training protocol and increased rubric specificity.


CONCLUSIONS
In general, results support the commonsense assumption that providing raters with detailed rubrics and comprehensive training will indeed improve measurement outcomes. Although the steps taken here were in the right direction, there remains room for improvement. Efforts are currently under way to further improve both the scoring rubrics and rater training.
Baldwin
Su G
SG
National Board of Medical Examiners, 3750 Market Street, Philadelphia, PA, 19104, USA. sbaldwin@nbme.org
Harik
Polina
P
Keller
Lisa A
LA
Clauser
Brian E
BE
Baldwin
Peter
P
Rebbecchi
Thomas A
TA
D002983:Clinical Competence; D004521:Educational Measurement / Q000379:methods* / Q000592:standards*; D008007:Licensure, Medical; D014481:United States