title: High-stakes assessment of the non-technical skills of critical care trainees using simulation: feasibility, acceptability and reliability.
abstract: OBJECTIVE
  To evaluate the use of high-fidelity simulation for summative high-stakes assessment of intensive care trainees, focusing on non-technical skills (NTS), testing feasibility and acceptability of simulation assessment, and the reliability of two NTS rating scales.
  
  
  METHODS
  Prospective observational study of senior intensive care trainees in a simulated specialist examination.
  
  
  METHODS
  Participants undertook a simulated patient management scenario and were assessed using two rating scales: the Anaesthesia Non-technical Skills (ANTS) scale and the Ottawa Global Rating Scale (GRS). Assessors were trained, currently active, high-stakes examiners. Participants also completed a survey on simulation-based summative assessment.
  
  
  METHODS
  The inter-rater reliability of two rating scales for NTS assessment. We evaluated the feasibility of simulation-based assessment, and used survey results to assess acceptability to participants.
  
  
  RESULTS
  Simulation assessment was feasible. Participants considered simulation-based high-stakes assessment to be acceptable and felt their scenario performance was reflective of real-world performance. Participants identified a need for debriefing following scenario-based assessment. Inter-rater reliability was fair for the ANTS and Ottawa GRS scores (intra-class correlation coefficient, 0.39 and 0.42, respectively). There was only fair agreement between raters for an NTS pass or fail (weighted kappa, 0.32) and for a technical skills pass or fail (weighted kappa, 0.36).
  
  
  CONCLUSIONS
  Summative high-stakes assessment using a single simulated scenario was feasible and acceptable to senior intensive care trainees. The low inter-rater reliability for the ANTS and Ottawa GRS rating scales and for pass or fail discrimination may limit its incorporation into an existing examination format.
authors.affiliation: University of Queensland, Brisbane, QLD, Australia. l.nunnink@uq.edu.au.
authors.forename: Leo
authors.initials: L
authors.lastname: Nunnink
authors.affiliation: University of Sydney, Sydney, NSW, Australia.
authors.forename: Carole
authors.initials: C
authors.lastname: Foot
authors.affiliation: University of Queensland, Brisbane, QLD, Australia.
authors.forename: Bala
authors.initials: B
authors.lastname: Venkatesh
authors.affiliation: Geelong Hospital, Geelong, VIC, Australia.
authors.forename: Charlie
authors.initials: C
authors.lastname: Corke
authors.affiliation: University of New South Wales, Sydney, NSW, Australia.
authors.forename: Manoj
authors.initials: M
authors.lastname: Saxena
authors.affiliation: University of Sydney, Sydney, NSW, Australia.
authors.forename: Mark
authors.initials: M
authors.lastname: Lucey
authors.affiliation: University of Queensland, Brisbane, QLD, Australia.
authors.forename: Mark
authors.initials: M
authors.lastname: Jones
mesh_terms: D002983:Clinical Competence; D003198:Computer Simulation; D003422:Critical Care; D004502:Education, Medical, Continuing / Q000379:methods*; D005178:Faculty / Q000592:standards*; D006801:Humans; D007396:Internship and Residency / Q000379:methods*; D011446:Prospective Studies; D015203:Reproducibility of Results