Faculty and resident evaluations of medical students on a surgery clerkship correlate poorly with standardized exam scores.
BACKGROUND
The clinical knowledge of medical students on a surgery clerkship is routinely assessed via subjective evaluations from faculty members and residents. Interpretation of these ratings should ideally be valid and reliable. However, prior literature has questioned the correlation between subjective and objective components when assessing students' clinical knowledge.


METHODS
Retrospective cross-sectional data were collected from medical student records at The Johns Hopkins University School of Medicine from July 2009 through June 2011. Surgical faculty members and residents rated students' clinical knowledge on a 5-point, Likert-type scale. Interrater reliability was assessed using intraclass correlation coefficients for students with ≥4 attending surgeon evaluations (n = 216) and ≥4 resident evaluations (n = 207). Convergent validity was assessed by correlating average evaluation ratings with scores on the National Board of Medical Examiners (NBME) clinical subject examination for surgery. Average resident and attending surgeon ratings were also compared by NBME quartile using analysis of variance.


RESULTS
There were high degrees of reliability for resident ratings (intraclass correlation coefficient, .81) and attending surgeon ratings (intraclass correlation coefficient, .76). Resident and attending surgeon ratings shared a moderate degree of variance (19%). However, average resident ratings and average attending surgeon ratings shared a small degree of variance with NBME surgery examination scores (ρ(2) ≤ .09). When ratings were compared among NBME quartile groups, the only significant difference was for residents' ratings of students with the lower 25th percentile of scores compared with the top 25th percentile of scores (P = .007).


CONCLUSIONS
Although high interrater reliability suggests that attending surgeons and residents rate students with consistency, the lack of convergent validity suggests that these ratings may not be reflective of actual clinical knowledge. Both faculty members and residents may benefit from training in knowledge assessment, which will likely increase opportunities to recognize deficiencies and make student evaluation a more valuable tool.
Department of Surgery, Johns Hopkins School of Medicine, 1800 Orleans Street, Bloomberg Children's Center 7310, Baltimore, MD 21287, USA. Electronic address: sgoldstein@jhmi.edu.
Seth D
SD
Goldstein
Department of Surgery, Johns Hopkins School of Medicine, 1800 Orleans Street, Bloomberg Children's Center 7310, Baltimore, MD 21287, USA.
Brenessa
B
Lindeman
Office of Medical Education Services, Johns Hopkins School of Medicine, Baltimore, MD, USA.
Jorie
J
Colbert-Getz
Department of Surgery, Johns Hopkins School of Medicine, 1800 Orleans Street, Bloomberg Children's Center 7310, Baltimore, MD 21287, USA.
Trisha
T
Arbella
Department of Pediatrics, Johns Hopkins School of Medicine, Baltimore, MD, USA.
Robert
R
Dudas
Department of Surgery, Johns Hopkins School of Medicine, 1800 Orleans Street, Bloomberg Children's Center 7310, Baltimore, MD 21287, USA.
Anne
A
Lidor
Department of Surgery, Johns Hopkins School of Medicine, 1800 Orleans Street, Bloomberg Children's Center 7310, Baltimore, MD 21287, USA.
Bethany
B
Sacks
D002982:Clinical Clerkship / Q000458:organization & administration*; D002983:Clinical Competence; D003430:Cross-Sectional Studies; D004521:Educational Measurement / Q000379:methods*; D005180:Faculty, Medical; D013502:General Surgery / Q000193:education*; D006801:Humans; D007396:Internship and Residency / Q000379:methods*; D011446:Prospective Studies; D015203:Reproducibility of Results; D013337:Students, Medical
Assessment; Medical student education; Surgery clerkship