Can skills coaches be used to assess resident performance in the skills laboratory?
BACKGROUND
The purpose of this study was to compare faculty ratings between live versus video-recorded resident performances and faculty versus skills coaches' ratings of video-recorded resident performances.


METHODS
PGY1 residents were observed, video-recorded, and rated during a Verification of Proficiency examination on 4 stations (ie, suturing, laparotomy, central line, and cricothyroidotomy). One surgeon and 2 trained skills coaches independently rated each video-recorded performance (N = 25). The chi-square test was used to compare checklist ratings. Analysis of variance was used to compare global ratings. Intraclass correlations were used to evaluate inter-rater agreement.


RESULTS
There were no statistical differences in faculty checklist ratings for live versus video-recorded performances (P > .05), and we found a nearly perfect interrater agreement, intraclass correlation coefficient (ICC) = 0.99 (P < .001). When comparing faculty versus skills coaches' ratings on video-recorded performances, we found no differences for the global or checklist ratings. Inter-rater agreement was moderately high for the global ratings, ICC = 0.71 (P <. 0.01, 95% confidence interval 0.23-0.96), and nearly perfect for the checklist ratings, ICC = 0.99 (P < .001, 95% confidence interval 0.94-1.00).


CONCLUSIONS
When assessing residents' performances, use of video-recorded performance ratings and skills coaches may be viable alternatives to live ratings performed by surgical faculty.
Northwestern Center for Advanced Surgical Education, Simulation Technology and Immersive Learning, Feinberg School of Medicine, Chicago, IL 60611, USA. d-rooney@northwestern.edu
Deborah M
DM
Rooney
Eric S
ES
Hungness
Debra A
DA
Darosa
Carla M
CM
Pugh
D057189:Checklist; D002983:Clinical Competence / Q000592:standards*; D003479:Curriculum / Q000592:standards; D004521:Educational Measurement / Q000379:methods*; D005180:Faculty, Medical; D013502:General Surgery / Q000193:education*; D006282:Health Personnel; D006801:Humans; D007396:Internship and Residency / Q000592:standards*; D015588:Observer Variation; D010820:Physicians; D013514:Surgical Procedures, Operative / Q000193:education*; D013663:Teaching / Q000379:methods*; D014741:Video Recording