title: Development of workplace-based assessments of non-technical skills in anaesthesia.
abstract: Non-technical skills are recognised as crucial to good anaesthetic practice. We designed and evaluated a specialty-specific tool to assess non-technical aspects of trainee performance in theatre, based on a system previously found reliable in a recruitment setting. We compared inter-rater agreement (multir-ater kappa) for live assessments in theatre with that in a selection centre and a video-based rater training exercise. Twenty-seven trainees participated in the first in-theatre assessment round and 40 in the second. Round- 1 scores had poor inter-rater agreement (mean kappa = 0.20) and low reliability (generalisability coefficient G = 0.50). A subsequent assessor training exercise showed good inter-rater agreement, (mean kappa = 0.79) but did not improve performance of the assessment tool when used in round 2 (mean kappa = 0.14, G = 0.42). Inter-rater agreement in two selection centres (mean kappa = 0.61 and 0.69) exceeded that found in theatre. Assessment tools that perform reliably in controlled settings may not do so in the workplace.
authors.affiliation: Department of Anaesthesia, Plymouth Hospitals NHS Trust, Plymouth, UK. gemmacrossingham@hotmail.com
authors.forename: G V
authors.initials: GV
authors.lastname: Crossingham
authors.forename: P J A
authors.initials: PJ
authors.lastname: Sice
authors.forename: M J
authors.initials: MJ
authors.lastname: Roberts
authors.forename: W H
authors.initials: WH
authors.lastname: Lam
authors.forename: T C E
authors.initials: TC
authors.lastname: Gale
mesh_terms: D000776:Anesthesiology / Q000193:education*; D000777:Anesthetics; D002983:Clinical Competence; D003627:Data Interpretation, Statistical; D004521:Educational Measurement; D006801:Humans; D015588:Observer Variation; D009873:Operating Rooms; D010820:Physicians; D011785:Quality Assurance, Health Care; D015203:Reproducibility of Results; D014741:Video Recording; D000078329:Workforce
chemical_list: D000777:Anesthetics