title: Can I leave the theatre? A key to more reliable workplace-based assessment.
abstract: BACKGROUND
  The value of workplace-based assessments such as the mini-clinical evaluation exercise (mini-CEX), and clinicians' confidence and engagement in the process, has been constrained by low reliability and limited capacity to identify underperforming trainees. We proposed that changing the way supervisors make judgements about trainees would improve score reliability and identification of underperformers. Anaesthetists regularly make decisions about the level of trainee independence with a case, based on how closely they need to supervise them. We therefore used this as the basis for a new scoring system.
  
  
  METHODS
  We analysed 338 mini-CEXs where supervisors scored trainees using the conventional system, and also scored trainee independence, based on the need for direct, or more distant, supervision. As supervisory requirements depend on case difficulty, we then compared the actual trainee independence score and the expected trainee independence score obtained externally.
  
  
  RESULTS
  Compared with the conventional scoring system used in previous studies, reliability was very substantially improved using a system based on a trainee's level of independence with a case. Reliability improved further when this score was corrected for case difficulty. Furthermore, the new scoring system overcame the previously identified problem of assessor leniency and identified a number of trainees performing below expectations.
  
  
  CONCLUSIONS
  Supervisors' judgements on trainee independence with a case, based on the need for direct or more distant supervision, can generate reliable scores of trainee ability without the need for an onerous number of assessments, identify trainees performing below expectations, and track trainee progress towards independent specialist practice.
authors.affiliation: Centre for Medical and Health Sciences Education, Faculty of Medical and Health Sciences, University of Auckland, 2 Park Rd, Grafton, Auckland 1010, New Zealand Department of Anaesthesia, Auckland City Hospital, 2 Park Rd, Grafton, Auckland 1010, New Zealand j.weller@auckland.ac.nz.
authors.forename: J M
authors.initials: JM
authors.lastname: Weller
authors.affiliation: Department of Anaesthesia, Auckland City Hospital, 2 Park Rd, Grafton, Auckland 1010, New Zealand.
authors.forename: M
authors.initials: M
authors.lastname: Misur
authors.affiliation: Department of Anaesthesia, Auckland City Hospital, 2 Park Rd, Grafton, Auckland 1010, New Zealand.
authors.forename: S
authors.initials: S
authors.lastname: Nicolson
authors.affiliation: Department of Anaesthesia, Royal Melbourne Hospital, Grattan Street, Parkville, VIC 3052, Australia.
authors.forename: J
authors.initials: J
authors.lastname: Morris
authors.affiliation: Department of Anaesthesia, Wellington Hospital, Riddiford Street, Newtown, Wellington 6021, New Zealand.
authors.forename: S
authors.initials: S
authors.lastname: Ure
authors.affiliation: Academic Unit of Medical Education, University of Sheffield, 85 Wilkinson Street, Sheffield S10 2GJ, UK.
authors.forename: J
authors.initials: J
authors.lastname: Crossley
authors.affiliation: University of Newcastle, University Drive, Callaghan, Newcastle, NSW 2308, Australia.
authors.forename: B
authors.initials: B
authors.lastname: Jolly
mesh_terms: D000776:Anesthesiology / Q000193:education* / Q000706:statistics & numerical data; D001315:Australia; D002983:Clinical Competence / Q000706:statistics & numerical data*; D004503:Education, Medical, Graduate / Q000379:methods* / Q000706:statistics & numerical data; D004521:Educational Measurement / Q000379:methods* / Q000706:statistics & numerical data; D006784:Hospitals, Teaching; D006801:Humans; D007600:Judgment / Q000502:physiology; D009520:New Zealand; D015203:Reproducibility of Results; D017132:Workplace / Q000706:statistics & numerical data*
keywords: educational assessment; educational measurement; medical education, graduate; reliability; workplace