Decreasing variability in faculty ratings of student case presentations: a faculty development intervention focusing on reflective practice.
BACKGROUND
Faculty perceptions of medical student knowledge are often based on case presentations, but their assessment of student performance can vary significantly based on content, presentation skill, and interpersonal characteristics.


OBJECTIVE
To determine if a brief faculty development intervention could significantly decrease (>or= 20%) faculty members' variability in rating student case presentations in the ambulatory setting.


METHODS
During the 1998-99 academic year, we videotaped 10 third-year medical students during one well-child presentation. Seven general pediatric faculty rated the videotapes using a validated scoring instrument measuring content: History of Present Illness (HPI), Past Medical History, and Physical Exam (PE); and communication skill: Precision of Language, Fluency, and Economy. Baseline ratings were from June to September 1999 and then repeated in October to November 2000 following a workshop where faculty reflected on the rating process and decreasing rating variance. The change in mean faculty ratings of the student's overall performance and for each item and the change in the mean difference in standard deviation before and after the workshop were analyzed.


RESULTS
Internal consistency of the scoring instrument using Cronbach's alpha was 0.88. The mean faculty rating for overall presentation performance and the PE did not significantly change after the workshop, whereas for all other items it did. The mean standard deviation of faculty ratings decreased significantly for the overall performance and all other items except HPI.


CONCLUSIONS
Despite using a validated scoring instrument to rate student case presentations significant variability exists among faculty, with rating inflation likely. A brief faculty development intervention using "reflection-on-action" was associated with more congruent ratings in the short term.
Ottolini
Mary C
MC
Department of Pediatric Medical Education, Children's National Medical Center, Washington, DC 20010, USA. Mottolin@cnmc.org
Cuzzi
Sandra
S
Tender
Jennifer
J
Coddington
Dale A
DA
Focht
Cheryl
C
Patel
Kantilal M
KM
Greenberg
Larrie
L
D004219:District of Columbia; D004521:Educational Measurement / Q000592:standards*; D005180:Faculty, Medical; D006801:Humans; D015588:Observer Variation; D013337:Students, Medical; D013850:Thinking; D014743:Videotape Recording