title: Use of Emergency Medicine Milestones as Items on End-of-Shift Evaluations Results in Overestimates of Residents' Proficiency Level.
abstract: BACKGROUND
  The emergency medicine milestones were developed to provide more objective resident assessment than current methods. However, little is known about the best practices for applying the milestones in resident assessment.
  
  
  OBJECTIVE
  We examined the utility of end-of-shift evaluations (ESEs) constructed using the milestones in resident assessment.
  
  
  METHODS
  We developed 14 daily ESEs, each of which included 9 or 10 emergency medicine milestones. Postgraduate year (PGY)-1 and PGY-2 residents were assessed on milestone levels 1 through 3; PGY-3 and PGY-4 residents were assessed on levels 3 through 5. Each milestone was rated on a nominal scale (yes, no, or not applicable). The Clinical Competency Committee combined the ESE data with data from other assessments to determine each resident's proficiency level for the emergency medicine subcompetencies. We used descriptive statistics to summarize resident ESEs and milestone levels. We analyzed differences in ESE score across PGY levels using t tests and analyses of variance.
  
  
  RESULTS
  Faculty completed 763 ESEs on 33 residents with a range of 2 to 54 (median=22) ESEs per resident. Faculty rarely (8%, 372 of 4633) rated a resident as not achieving a milestone on the ESEs. Analyses of variance revealed that ESE scores on level 3 milestones did not differ significantly by PGY level. There was poor agreement between ESE scores and Clinical Competency Committee ratings.
  
  
  CONCLUSIONS
  The ESEs constructed using the milestones resulted in grade or milestone inflation. Our results do not support using milestones as a stand-alone assessment tool.
authors.forename: Erin
authors.initials: E
authors.lastname: Dehon
authors.forename: Jonathan
authors.initials: J
authors.lastname: Jones
authors.forename: Michael
authors.initials: M
authors.lastname: Puskarich
authors.forename: John Petty
authors.initials: JP
authors.lastname: Sandifer
authors.forename: Kristina
authors.initials: K
authors.lastname: Sikes
mesh_terms: D002983:Clinical Competence; D004521:Educational Measurement / Q000379:methods*; D004635:Emergency Medicine / Q000193:education*; D006801:Humans; D007396:Internship and Residency / Q000458:organization & administration*; D012189:Retrospective Studies