title: External validation of simulation-based assessments with other performance measures of third-year anesthesiology residents.
abstract: BACKGROUND
  There has been interest in the use of high-fidelity medical simulation to evaluate performance. We hypothesized that technical and nontechnical performance in the simulated environment is related to other various criterion measures, providing evidence to support the validity of the scores from the performance-based assessment.
  
  
  METHODS
  Twelve third-year anesthesia residents participated in a series of 6 short 5-minute scenarios and 1 longer 30-minute scenario. The short scenarios measured technical skills, whereas the longer one focused on nontechnical skills. Two independent raters scored subjects using analytic and holistic ratings. Short scenarios involved acute hemorrhage, blocked endotracheal tube, bronchospasm, hyperkalemia, tension pneumothorax, and unstable ventricular tachycardia. The long scenario concerned management of myocardial ischemia/infarction leading to cardiac arrest. Scores from the simulations were correlated with (a) rankings generated from an Internet-based global ranking instrument that categorized residents based on overall clinical ability and (b) residency board scores.
  
  
  RESULTS
  There were moderate correlations between various participant scores from the simulation-based assessment and aggregate rankings based on the global ranking instrument and residency examination scores.
  
  
  CONCLUSIONS
  The associations between simulator performance, both for technical and nontechnical skills, and other markers of ability provide some evidence to support the validity of simulation-based assessment scores. Replication studies with larger numbers of residents are warranted.
authors.affiliation: Anesthesia Service, VA Palo Alto Health Care System, Palo Alto, CA 94304, USA. mudumbai@stanford.edu
authors.forename: Seshadri C
authors.initials: SC
authors.lastname: Mudumbai
authors.forename: David M
authors.initials: DM
authors.lastname: Gaba
authors.forename: John R
authors.initials: JR
authors.lastname: Boulet
authors.forename: Steven K
authors.initials: SK
authors.lastname: Howard
authors.forename: M Frances
authors.initials: MF
authors.lastname: Davies
mesh_terms: D000328:Adult; D000776:Anesthesiology / Q000193:education*; D002983:Clinical Competence; D004503:Education, Medical, Graduate / Q000379:methods*; D004521:Educational Measurement / Q000379:methods; D004522:Educational Status; D005180:Faculty, Medical; D007722:Health Knowledge, Attitudes, Practice; D006801:Humans; D007396:Internship and Residency; D007858:Learning; D015203:Reproducibility of Results; D013223:Statistics as Topic; D013663:Teaching / Q000379:methods*; D013997:Time Factors