The generalizability of documentation scores from the USMLE Step 2 Clinical Skills examination.
BACKGROUND
This research examined various sources of measurement error in the documentation score component of the United States Medical Licensing Examination (USMLE) Step 2 Clinical Skills examination.


METHODS
A generalizability theory framework was employed to examine the documentation ratings for 847 examinees who completed the USMLE Step 2 Clinical Skills examination during an eight-day period in 2006. Each patient note was scored by two different raters allowing for a persons-crossed-with-raters-nested-in-cases design.


RESULTS
The results suggest that inconsistent performance on the part of raters makes a substantially greater contribution to measurement error than case specificity. Double scoring the notes significantly increases precision.


CONCLUSIONS
The results provide guidance for improving operational scoring of the patient notes. Double scoring of the notes may produce an increase in the precision of measurement equivalent to that achieved by lengthening the test by more than 50%. The study also cautions researchers that when examining sources of measurement error, inappropriate data-collection designs may result in inaccurate inferences.
Clauser
Brian E
BE
National Board of Medical Examiners, 3750 Market Street, Philadelphia, PA 19104, USA. bclauser@nbme.org
Harik
Polina
P
Margolis
Melissa J
MJ
Mee
Janet
J
Swygert
Kimberly
K
Rebbecchi
Thomas
T
D002983:Clinical Competence; D015331:Cohort Studies; D003142:Communication; D005793:Generalization, Psychological; D006801:Humans; D008007:Licensure, Medical; D015588:Observer Variation; D016544:Patient Simulation; D010808:Physical Examination; D010817:Physician-Patient Relations; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D014481:United States