title: A rater training protocol to assess team performance.
abstract: BACKGROUND
  Simulation-based methodologies are increasingly used to assess teamwork and communication skills and provide team training. Formative feedback regarding team performance is an essential component. While effective use of simulation for assessment or training requires accurate rating of team performance, examples of rater-training programs in health care are scarce. We describe our rater training program and report interrater reliability during phases of training and independent rating.
  
  
  METHODS
  We selected an assessment tool shown to yield valid and reliable results and developed a rater training protocol with an accompanying rater training handbook. The rater training program was modeled after previously described high-stakes assessments in the setting of 3 facilitated training sessions. Adjacent agreement was used to measure interrater reliability between raters.
  
  
  RESULTS
  Nine raters with a background in health care and/or patient safety evaluated team performance of 42 in-situ simulations using post-hoc video review. Adjacent agreement increased from the second training session (83.6%) to the third training session (85.6%) when evaluating the same video segments. Adjacent agreement for the rating of overall team performance was 78.3%, which was added for the third training session. Adjacent agreement was 97% 4 weeks posttraining and 90.6% at the end of independent rating of all simulation videos.
  
  
  CONCLUSIONS
  Rater training is an important element in team performance assessment, and providing examples of rater training programs is essential. Articulating key rating anchors promotes adequate interrater reliability. In addition, using adjacent agreement as a measure allows differentiation between high- and low-performing teams on video review.
authors.forename: Walter
authors.initials: W
authors.lastname: Eppich
authors.forename: Anna P
authors.initials: AP
authors.lastname: Nannicelli
authors.forename: Nicholas P
authors.initials: NP
authors.lastname: Seivert
authors.forename: Min-Woong
authors.initials: MW
authors.lastname: Sohn
authors.forename: Ranna
authors.initials: R
authors.lastname: Rozenfeld
authors.forename: Donna M
authors.initials: DM
authors.lastname: Woods
authors.forename: Jane L
authors.initials: JL
authors.lastname: Holl
mesh_terms: D004650:Employee Performance Appraisal / Q000592:standards*; D006801:Humans; D010348:Patient Care Team / Q000592:standards*; D061214:Patient Safety; D000066908:Simulation Training; D000070260:Teacher Training / Q000458:organization & administration*
keywords: assessment; health services research; multisource feedback/peer assessment; program planning/curriculum development; rater training; simulation; team performance