title: Making robust assessments of specialist trainees' workplace performance.
abstract: BACKGROUND
  Workplace-based assessments should provide a reliable measure of trainee performance, but have met with mixed success. We proposed that using an entrustability scale, where supervisors scored trainees on the level of supervision required for the case would improve the utility of compulsory mini-clinical evaluation exercise (CEX) assessments in a large anaesthesia training program.
  
  
  METHODS
  We analysed mini-CEX scores from all Australian and New Zealand College of Anaesthetists trainees submitted to an online database over a 12-month period. Supervisors' scores were adjusted for the expected supervision requirement for the case for trainees at different stages of training. We used generalisability theory to determine score reliability.
  
  
  RESULTS
  7808 assessments were available for analysis. Supervision requirements decreased significantly (P < 0.05) with increased duration and level of training, supporting validity. We found moderate reliability (G > 0.7) with a feasible number of assessments. Adjusting scores against the expected supervision requirement considerably improved reliability, with G > 0.8 achieved with only nine assessments. Three per cent of trainees generated average mini-CEX scores below the expected standard.
  
  
  CONCLUSIONS
  Using an entrustment scoring system, where supervisors score trainees on the level of supervision required, mini-CEX scores demonstrated moderate reliability within a feasible number of assessments, and evidence of validity. When scores were adjusted against an expected standard, underperforming trainees could be identified, and reliability much improved. Taken together with other evidence on trainee ability, the mini-CEX is of sufficient reliability for inclusion in high stakes decisions on trainee progression towards independent specialist practice.
authors.affiliation: Centre for Medical and Health Sciences Education, School of Medicine, University of Auckland, New Zealand j.weller@auckland.ac.nz.|Department of Anaesthesia, Auckland City Hospital, New Zealand.
authors.forename: J M
authors.initials: JM
authors.lastname: Weller
authors.affiliation: Department of Anaesthesia and Perioperative Medicine, Monash Health, Victoria, Australia.|Department of Anaesthesia and Perioperative Medicine, Monash University, Clayton, Victoria, Australia.
authors.forename: D J
authors.initials: DJ
authors.lastname: Castanelli
authors.affiliation: Centre for Medical and Health Sciences Education, School of Medicine, University of Auckland, New Zealand.
authors.forename: Y
authors.initials: Y
authors.lastname: Chen
authors.affiliation: Medical Education Unit, School of Medicine and Public Health, Faculty of Health and Medicine, University of Newcastle, New South Wales, Australia.
authors.forename: B
authors.initials: B
authors.lastname: Jolly
mesh_terms: D000776:Anesthesiology / Q000193:education*; D004521:Educational Measurement; D006801:Humans; D013038:Specialization; D017132:Workplace
keywords: educational assessment; educational measurement; graduate; medical education; reliability; workplace