title: Initial performance of a modified milestones global evaluation tool for semiannual evaluation of residents by faculty.
abstract: OBJECTIVE
  To determine whether faculty could successfully evaluate residents using a competency-based modified Milestones global evaluation tool.
  
  
  METHODS
  A program's leadership team modified a draft Surgery Milestones Working Group summative global assessment instrument into a modified Milestones tool (MMT) for local use during faculty meetings devoted to semiannual resident review. Residents were scored on 15 items spanning all competencies using an 8-point graphic response scale; unstructured comments also were solicited. Arithmetic means were computed at the resident and postgraduate year cohort levels for items and competency item sets. Score ranges (highest minus lowest score) were calculated; variability was termed "low" (range <2.0 points), "moderate" (range = 2.0), or "high" (range >2.0). A subset of "low" was designated "small" (1.0-1.9). Trends were sought among item, competency, and total Milestones scores. MMT correlations with examination scores and multisource (360°) assessments were explored. The success of implementing MMT was judged using published criteria for educational assessment methods.
  
  
  METHODS
  Fully accredited, independently sponsored residency.
  
  
  METHODS
  Program leaders and 22 faculty members (71% voluntary, mean 12y of experience).
  
  
  RESULTS
  Twenty-six residents were assessed, yielding 7 to 13 evaluations for MMT per categorical resident and 3 to 6 per preliminary trainee. Scores spanned the entire response scale. All MMT evaluations included narrative comments. Individual resident score variability was low (96% within competencies and 92% across competencies). Subset analysis showed that small variations were common (35% within competencies and 54% across competencies). Postgraduate year cohort variability was higher (61% moderate or high within competencies and 50% across competencies). Cohort scores at the item, competency, and total score levels exhibited rising trajectories, suggesting MMT construct validity. MMT scores did not demonstrate concurrent validity, correlating poorly with other metrics. The MMT met multiple criteria for good assessment.
  
  
  CONCLUSIONS
  A modified Milestones global evaluation tool can be successfully adopted for semiannual assessments of resident performance by volunteer faculty members.
authors.affiliation: Department of Surgery, Abington Memorial Hospital, Abington, Pennsylvania. Electronic address: Karen.R.Borman@medstar.net.
authors.forename: Karen R
authors.initials: KR
authors.lastname: Borman
authors.forename: Rebecca
authors.initials: R
authors.lastname: Augustine
authors.forename: Thomas
authors.initials: T
authors.lastname: Leibrandt
authors.forename: Christopher M
authors.initials: CM
authors.lastname: Pezzi
authors.forename: John S
authors.initials: JS
authors.lastname: Kukora
mesh_terms: D000328:Adult; D002983:Clinical Competence; D004503:Education, Medical, Graduate / Q000458:organization & administration; D004521:Educational Measurement / Q000379:methods*; D005069:Evaluation Studies as Topic; D005180:Faculty, Medical / Q000592:standards*; D005260:Female; D006801:Humans; D007396:Internship and Residency / Q000458:organization & administration*; D008297:Male; D018794:Problem-Based Learning; D012680:Sensitivity and Specificity; D013997:Time Factors; D014481:United States
keywords: Interpersonal and Communication Skills; Medical Knowledge; Patient Care; Practice-Based Learning and Improvement; Systems-Based Practice; competency-based education; graduate medical education; milestones; resident assessment