title: Duration of faculty training needed to ensure reliable or performance ratings.
abstract: OBJECTIVE
  The American Board of Surgery has mandated intraoperative assessment of general surgery residents, yet the time required to train faculty to accurately and reliably complete operating room performance evaluation forms is unknown. Outside of surgical education, frame-of-reference (FOR) training has been shown to be an effective training modality to teach raters the specific performance indicators associated with each point on a rating scale. Little is known, however, about what form and duration of FOR training is needed to accomplish reliable ratings among surgical faculty.
  
  
  METHODS
  Two groups of surgical faculty separately underwent either an accelerated 1-hour (n = 10) or immersive four-hour (n = 34) FOR faculty development program. Both programs included a formal presentation and a facilitated discussion of sample behaviors for each point on the Zwisch operating room performance rating scale (see DaRosa et al.(8)). The immersive group additionally participated in a small group exercise that included additional practice. After training, both groups were tested using 10 video clips of trainees at various levels. Responses were scored against expert consensus ratings. The 2-sided Mann-Whitney U test was used to compare between group means.
  
  
  METHODS
  All trainees were faculty members in the Department of Surgery of a large midwestern private medical school.
  
  
  RESULTS
  Faculty undergoing the 1-hour FOR training program did not have a statistically different mean correct response rate on the video test when compared with those undergoing the 4-hour training program (88% vs 80%; p = 0.07).
  
  
  CONCLUSIONS
  One-hour FOR training sessions are likely sufficient to train surgical faculty to reliably use a simple evaluation instrument for the assessment of intraoperative performance. Additional research is needed to determine how these results generalize to different assessment instruments.
authors.affiliation: Department of Surgery, Massachusetts General Hospital, Harvard Medical School, Boston, Massachusetts. Electronic address: bcgeorge@partners.org.
authors.forename: Brian C
authors.initials: BC
authors.lastname: George
authors.forename: Ezra N
authors.initials: EN
authors.lastname: Teitelbaum
authors.forename: Debra A
authors.initials: DA
authors.lastname: Darosa
authors.forename: Eric S
authors.initials: ES
authors.lastname: Hungness
authors.forename: Shari L
authors.initials: SL
authors.lastname: Meyerson
authors.forename: Jonathan P
authors.initials: JP
authors.lastname: Fryer
authors.forename: Mary
authors.initials: M
authors.lastname: Schuller
authors.forename: Joseph B
authors.initials: JB
authors.lastname: Zwischenberger
mesh_terms: D000328:Adult; D002983:Clinical Competence; D003198:Computer Simulation; D003479:Curriculum; D004503:Education, Medical, Graduate / Q000458:organization & administration; D004521:Educational Measurement; D005180:Faculty, Medical / Q000458:organization & administration*; D005260:Female; D013502:General Surgery / Q000193:education*; D006801:Humans; D007396:Internship and Residency / Q000458:organization & administration*; D008297:Male; D008875:Middle Aged; D009873:Operating Rooms; D015397:Program Evaluation; D058996:Quality Improvement; D018709:Statistics, Nonparametric; D013997:Time Factors; D014481:United States
keywords: Medical Knowledge; evaluation; faculty development; frame-of-reference training; measurement; rater training; surgical education