title: Decreasing variability in faculty ratings of student case presentations: a faculty development intervention focusing on reflective practice.
abstract: BACKGROUND
  Faculty perceptions of medical student knowledge are often based on case presentations, but their assessment of student performance can vary significantly based on content, presentation skill, and interpersonal characteristics.
  
  
  OBJECTIVE
  To determine if a brief faculty development intervention could significantly decrease (>or= 20%) faculty members' variability in rating student case presentations in the ambulatory setting.
  
  
  METHODS
  During the 1998-99 academic year, we videotaped 10 third-year medical students during one well-child presentation. Seven general pediatric faculty rated the videotapes using a validated scoring instrument measuring content: History of Present Illness (HPI), Past Medical History, and Physical Exam (PE); and communication skill: Precision of Language, Fluency, and Economy. Baseline ratings were from June to September 1999 and then repeated in October to November 2000 following a workshop where faculty reflected on the rating process and decreasing rating variance. The change in mean faculty ratings of the student's overall performance and for each item and the change in the mean difference in standard deviation before and after the workshop were analyzed.
  
  
  RESULTS
  Internal consistency of the scoring instrument using Cronbach's alpha was 0.88. The mean faculty rating for overall presentation performance and the PE did not significantly change after the workshop, whereas for all other items it did. The mean standard deviation of faculty ratings decreased significantly for the overall performance and all other items except HPI.
  
  
  CONCLUSIONS
  Despite using a validated scoring instrument to rate student case presentations significant variability exists among faculty, with rating inflation likely. A brief faculty development intervention using "reflection-on-action" was associated with more congruent ratings in the short term.
authors.affiliation: Department of Pediatric Medical Education, Children's National Medical Center, Washington, DC 20010, USA. Mottolin@cnmc.org
authors.forename: Mary C
authors.initials: MC
authors.lastname: Ottolini
authors.forename: Sandra
authors.initials: S
authors.lastname: Cuzzi
authors.forename: Jennifer
authors.initials: J
authors.lastname: Tender
authors.forename: Dale A
authors.initials: DA
authors.lastname: Coddington
authors.forename: Cheryl
authors.initials: C
authors.lastname: Focht
authors.forename: Kantilal M
authors.initials: KM
authors.lastname: Patel
authors.forename: Larrie
authors.initials: L
authors.lastname: Greenberg
mesh_terms: D004219:District of Columbia; D004521:Educational Measurement / Q000592:standards*; D005180:Faculty, Medical; D006801:Humans; D015588:Observer Variation; D013337:Students, Medical; D013850:Thinking; D014743:Videotape Recording