title: Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014.
abstract: US internal medicine residency programs are now required to rate residents using milestones. Evidence of validity of milestone ratings is needed.
  
  
  
  To compare ratings of internal medicine residents using the pre-2015 resident annual evaluation summary (RAES), a nondevelopmental rating scale, with developmental milestone ratings.
  
  
  
  Cross-sectional study of US internal medicine residency programs in the 2013-2014 academic year, including 21 284 internal medicine residents (7048 postgraduate-year 1 [PGY-1], 7233 PGY-2, and 7003 PGY-3).
  
  
  
  Program director ratings on the RAES and milestone ratings.
  
  
  
  Correlations of RAES and milestone ratings by training year; correlations of medical knowledge ratings with American Board of Internal Medicine (ABIM) certification examination scores; rating of unprofessional behavior using the 2 systems.
  
  
  
  Corresponding RAES ratings and milestone ratings showed progressively higher correlations across training years, ranging among competencies from 0.31 (95% CI, 0.29 to 0.33) to 0.35 (95% CI, 0.33 to 0.37) for PGY-1 residents to 0.43 (95% CI, 0.41 to 0.45) to 0.52 (95% CI, 0.50 to 0.54) for PGY-3 residents (all P values <.05). Linear regression showed ratings differed more between PGY-1 and PGY-3 years using milestone ratings than the RAES (all P values <.001). Of the 6260 residents who attempted the certification examination, the 618 who failed had lower ratings using both systems for medical knowledge than did those who passed (RAES difference, -0.9; 95% CI, -1.0 to -0.8; P < .001; milestone medical knowledge 1 difference, -0.3; 95% CI, -0.3 to -0.3; P < .001; and medical knowledge 2 difference, -0.2; 95% CI, -0.3 to -0.2; P < .001). Of the 26 PGY-3 residents with milestone ratings indicating deficiencies on either of the 2 medical knowledge subcompetencies, 12 failed the certification examination. Correlation of RAES ratings for professionalism with residents' lowest professionalism milestone ratings was 0.44 (95% CI, 0.43 to 0.45; P < .001).
  
  
  
  Among US internal medicine residents in the 2013-2014 academic year, milestone-based ratings correlated with RAES ratings but with a greater difference across training years. Both rating systems for medical knowledge correlated with ABIM certification examination scores. Milestone ratings may better detect problems with professionalism. These preliminary findings may inform establishment of the validity of milestone-based assessment.
authors.affiliation: University of California at San Francisco.
authors.forename: Karen E
authors.initials: KE
authors.lastname: Hauer
authors.affiliation: American Board of Internal Medicine, Philadelphia, Pennsylvania.
authors.forename: Jonathan
authors.initials: J
authors.lastname: Vandergrift
authors.affiliation: Hess Consulting, Lévis, Québec, Canada.
authors.forename: Brian
authors.initials: B
authors.lastname: Hess
authors.affiliation: American Board of Internal Medicine, Philadelphia, Pennsylvania.
authors.forename: Rebecca S
authors.initials: RS
authors.lastname: Lipner
authors.affiliation: Accreditation Council for Graduate Medical Education, Chicago, Illinois.
authors.forename: Eric S
authors.initials: ES
authors.lastname: Holmboe
authors.affiliation: American Board of Internal Medicine, Philadelphia, Pennsylvania.
authors.forename: Sarah
authors.initials: S
authors.lastname: Hood
authors.affiliation: Commonwealth Medical College, Scranton, Pennsylvania.
authors.forename: William
authors.initials: W
authors.lastname: Iobst
authors.affiliation: Accreditation Council for Graduate Medical Education, Chicago, Illinois.
authors.forename: Stanley J
authors.initials: SJ
authors.lastname: Hamstra
authors.affiliation: American Board of Internal Medicine, Philadelphia, Pennsylvania.
authors.forename: Furman S
authors.initials: FS
authors.lastname: McDonald
mesh_terms: D000328:Adult; D002568:Certification / Q000592:standards*; D002983:Clinical Competence / Q000706:statistics & numerical data*; D004521:Educational Measurement; D005260:Female; D006801:Humans; D007388:Internal Medicine / Q000193:education*; D007396:Internship and Residency / Q000706:statistics & numerical data*; D008297:Male; D028743:Professional Misconduct; D013044:Specialty Boards; D014481:United States