title: Competency Evaluations in the Next Accreditation System: Contributing to Guidelines and Implications.
abstract: CONSTRUCT: This study examines validity evidence of end-of-rotation evaluation scores used to measure competencies and milestones as part of the Next Accreditation System (NAS) of the Accreditation Council for Graduate Medical Education (ACGME).
  
  
  BACKGROUND
  Since the implementation of the milestones, end-of-rotation evaluations have surfaced as a potentially useful assessment method. However, validity evidence on the use of rotation evaluation scores as part of the NAS has not been studied. This article examines validity evidence for end-of-rotation evaluations that can contribute to developing guidelines that support the NAS.
  
  
  METHODS
  Data from 2,701 end-of-rotation evaluations measuring 21 out of 22 Internal Medicine milestones for 142 residents were analyzed (July 2013-June 2014). Descriptive statistics were used to measure the distribution of ratings by evaluators (faculty, n = 116; fellows, n = 59; peer-residents, n = 131), by postgraduate years. Generalizability analysis and higher order confirmatory factor analysis were used to examine the internal structure of ratings. Psychometric implications for combining evaluation scores using composite score reliability were examined.
  
  
  RESULTS
  Milestone ratings were significantly higher for each subsequent year of training (15/21 milestones). Faculty evaluators had greater variability in ratings across milestones, compared to fellows and residents; faculty ratings were generally correlated with milestone ratings from fellows (r = .45) and residents (r = .25), but lower correlations were found for Professionalism and Interpersonal and Communication Skills. The Φ-coefficient was .71, indicating good reliability. Internal structure supported a 6-factor solution, corresponding to the hierarchical relationship between the milestones and the 6 core competencies. Evaluation scores corresponding to Patient Care, Medical Knowledge, and Practice-Based Learning and Improvement had higher correlations to milestones reported to the ACGME. Mean evaluation ratings predicted problem residents (odds ratio = 5.82, p < .001).
  
  
  CONCLUSIONS
  Guidelines for rotation evaluations proposed in this study provide useful solutions that can help program directors make decisions on resident progress and contribute to assessment systems in graduate medical education.
authors: a Department of Medical Education , University of Illinois at Chicago College of Medicine , Chicago , Illinois , USA.
authors: Yoon Soo
authors: YS
authors: Park
authors: b Department of Medicine , University of Illinois at Chicago College of Medicine , Chicago , Illinois , USA.
authors: Fred A
authors: FA
authors: Zar
authors: c Foundation for Advancement of International Medical Education and Research , Philadelphia , Pennsylvania , USA.
authors: John J
authors: JJ
authors: Norcini
authors: a Department of Medical Education , University of Illinois at Chicago College of Medicine , Chicago , Illinois , USA.
authors: Ara
authors: A
authors: Tekian
mesh_terms: D000068:Accreditation; D002983:Clinical Competence / Q000592:standards*; D004503:Education, Medical, Graduate / Q000592:standards*; D004521:Educational Measurement / Q000379:methods*; D005163:Factor Analysis, Statistical; D005260:Female; D017408:Guidelines as Topic / Q000592:standards; D006801:Humans; D007388:Internal Medicine / Q000193:education*; D007396:Internship and Residency; D008297:Male; D011594:Psychometrics; D012189:Retrospective Studies
keywords: ACGME; end-of-rotation evaluations; milestones; next accreditation system; workplace-based assessments