title: Improving resident performance assessment data: numeric precision and narrative specificity.
abstract: OBJECTIVE
  To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments.
  
  
  METHODS
  Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments.
  
  
  RESULTS
  Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%).
  
  
  CONCLUSIONS
  Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.
authors.affiliation: University of Texas Health Science Center at San Antonio, Academic Informatics Services, 78229-3900, USA. littlefield@uthscsa.edu
authors.forename: John H
authors.initials: JH
authors.lastname: Littlefield
authors.forename: Debra A
authors.initials: DA
authors.lastname: Darosa
authors.forename: Judy
authors.initials: J
authors.lastname: Paukert
authors.forename: Reed G
authors.initials: RG
authors.lastname: Williams
authors.forename: Debra L
authors.initials: DL
authors.lastname: Klamen
authors.forename: John D
authors.initials: JD
authors.lastname: Schoolfield
mesh_terms: D000328:Adult; D002983:Clinical Competence; D013502:General Surgery / Q000193:education*; D006801:Humans; D007396:Internship and Residency / Q000592:standards; D008486:Physician Executives; D011785:Quality Assurance, Health Care; D013647:Task Performance and Analysis