title: Validity and reliability assessment of detailed scoring checklists for use during perioperative emergency simulation training.
abstract: BACKGROUND
  Few valid and reliable grading checklists have been published for the evaluation of performance during simulated high-stakes perioperative event management. As such, the purposes of this study were to construct valid scoring checklists for a variety of perioperative emergencies and to determine the reliability of scores produced by these checklists during continuous video review.
  
  
  METHODS
  A group of anesthesiologists, intensivists, and educators created a set of simulation grading checklists for the assessment of the following scenarios: severe anaphylaxis, cerebrovascular accident, hyperkalemic arrest, malignant hyperthermia, and acute coronary syndrome. Checklist items were coded as critical or noncritical. Nonexpert raters evaluated 10 simulation videos in a random order, with each video being graded 4 times. A group of faculty experts also graded the videos to create a reference standard to which nonexpert ratings were compared. P < 0.05 was considered significant.
  
  
  RESULTS
  Team leaders in the simulation videos were scored by the expert panel as having performed 56.5% of all items on the checklist (range, 43.8%-84.0%), and 67.2% of the critical items (range, 30.0%-100%). Nonexpert raters agreed with the expert assessment 89.6% of the time (95% confidence interval, 87.2%-91.6%). No learning curve development was found with repetitive video assessment or checklist use. The κ values comparing nonexpert rater assessments to the reference standard averaged 0.76 (95% confidence interval, 0.71-0.81).
  
  
  CONCLUSIONS
  The findings indicate that the grading checklists described are valid, are reliable, and could be used in perioperative crisis management assessment.
authors.affiliation: From the Department of Anesthesiology (M.D.M.), Vanderbilt University Medical Center, Nashville, TN; Departments of Anesthesia and Perioperative Medicine (W.R.H., C.M.F., L.C.F., C.A.C.), and Public Health Sciences (P.J.N.), Medical University of South Carolina, Charleston, SC; Department of Anesthesiology (V.K.M.), Columbia University Medical Center, New York, NY; and Section of Critical Care Medicine (M.F.O.) Department of Anesthesia and Critical Care (M.E.N.), University of Chicago, Chicago, IL.
authors.forename: Matthew D
authors.initials: MD
authors.lastname: McEvoy
authors.forename: William R
authors.initials: WR
authors.lastname: Hand
authors.forename: Cory M
authors.initials: CM
authors.lastname: Furse
authors.forename: Larry C
authors.initials: LC
authors.lastname: Field
authors.forename: Carlee A
authors.initials: CA
authors.lastname: Clark
authors.forename: Vivek K
authors.initials: VK
authors.lastname: Moitra
authors.forename: Paul J
authors.initials: PJ
authors.lastname: Nietert
authors.forename: Michael F
authors.initials: MF
authors.lastname: O'Connor
authors.forename: Mark E
authors.initials: ME
authors.lastname: Nunnally
mesh_terms: D057189:Checklist / Q000592:standards*; D002983:Clinical Competence / Q000592:standards*; D003198:Computer Simulation; D004632:Emergency Medical Services; D006801:Humans; D019990:Perioperative Care / Q000193:education*; D015203:Reproducibility of Results