title: Training less-experienced faculty improves reliability of skills assessment in cardiac surgery.
abstract: OBJECTIVE
  Previous work has demonstrated high inter-rater reliability in the objective assessment of simulated anastomoses among experienced educators. We evaluated the inter-rater reliability of less-experienced educators and the impact of focused training with a video-embedded coronary anastomosis assessment tool.
  
  
  METHODS
  Nine less-experienced cardiothoracic surgery faculty members from different institutions evaluated 2 videos of simulated coronary anastomoses (1 by a medical student and 1 by a resident) at the Thoracic Surgery Directors Association Boot Camp. They then underwent a 30-minute training session using an assessment tool with embedded videos to anchor rating scores for 10 components of coronary artery anastomosis. Afterward, they evaluated 2 videos of a different student and resident performing the task. Components were scored on a 1 to 5 Likert scale, yielding an average composite score. Inter-rater reliabilities of component and composite scores were assessed using intraclass correlation coefficients (ICCs) and overall pass/fail ratings with kappa.
  
  
  RESULTS
  All components of the assessment tool exhibited improvement in reliability, with 4 (bite, needle holder use, needle angles, and hand mechanics) improving the most from poor (ICC range, 0.09-0.48) to strong (ICC range, 0.80-0.90) agreement. After training, inter-rater reliabilities for composite scores improved from moderate (ICC, 0.76) to strong (ICC, 0.90) agreement, and for overall pass/fail ratings, from poor (kappa = 0.20) to moderate (kappa = 0.78) agreement.
  
  
  CONCLUSIONS
  Focused, video-based anchor training facilitates greater inter-rater reliability in the objective assessment of simulated coronary anastomoses. Among raters with less teaching experience, such training may be needed before objective evaluation of technical skills.
authors.affiliation: Northwestern University, Chicago, Ill.
authors.forename: Xiaoying
authors.initials: X
authors.lastname: Lou
authors.affiliation: St Louis University, St Louis, Mo.
authors.forename: Richard
authors.initials: R
authors.lastname: Lee
authors.affiliation: University of North Carolina at Chapel Hill, Chapel Hill, NC.
authors.forename: Richard H
authors.initials: RH
authors.lastname: Feins
authors.affiliation: Northwestern University, Chicago, Ill.
authors.forename: Daniel
authors.initials: D
authors.lastname: Enter
authors.affiliation: University of Rochester Medical Center, Rochester, NY.
authors.forename: George L
authors.initials: GL
authors.lastname: Hicks
authors.affiliation: University of Washington Medical Center, Seattle, Wash.
authors.forename: Edward D
authors.initials: ED
authors.lastname: Verrier
authors.affiliation: Stanford University, Stanford, Calif. Electronic address: jfann@stanford.edu.
authors.forename: James I
authors.initials: JI
authors.lastname: Fann
mesh_terms: D000714:Anastomosis, Surgical; D002983:Clinical Competence; D001026:Coronary Artery Bypass / Q000193:education* / Q000592:standards; D003331:Coronary Vessels / Q000601:surgery*; D004503:Education, Medical, Graduate / Q000379:methods* / Q000592:standards; D004504:Education, Medical, Undergraduate / Q000379:methods* / Q000592:standards; D004521:Educational Measurement / Q000379:methods* / Q000592:standards; D005180:Faculty, Medical; D005260:Female; D006801:Humans; D007396:Internship and Residency; D059032:Learning Curve; D008297:Male; D008953:Models, Anatomic; D008955:Models, Cardiovascular; D015588:Observer Variation; D015203:Reproducibility of Results; D013337:Students, Medical; D013647:Task Performance and Analysis; D014741:Video Recording