title: Reliability and validity of procedure-based assessments in otolaryngology training.
abstract: OBJECTIVE
  To investigate the reliability and construct validity of procedure-based assessment (PBA) in assessing performance and progress in otolaryngology training.
  
  
  METHODS
  Retrospective database analysis using a national electronic database.
  
  
  METHODS
  We analyzed PBAs of otolaryngology trainees in North London from core trainees (CTs) to specialty trainees (STs). The tool contains six multi-item domains: consent, planning, preparation, exposure/closure, technique, and postoperative care, rated as "satisfactory" or "development required," in addition to an overall performance rating (pS) of 1 to 4. Individual domain score, overall calculated score (cS), and number of "development-required" items were calculated for each PBA. Receiver operating characteristic analysis helped determine sensitivity and specificity.
  
  
  RESULTS
  There were 3,152 otolaryngology PBAs from 46 otolaryngology trainees analyzed. PBA reliability was high (Cronbach's α 0.899), and sensitivity approached 99%. cS correlated positively with pS and level in training (rs : +0.681 and +0.324, respectively). ST had higher cS and pS than CT (93% ± 0.6 and 3.2 ± 0.03 vs. 71% ± 3.1 and 2.3 ± 0.08, respectively; P < .001). cS and pS increased from CT1 to ST8 showing construct validity (rs : +0.348 and +0.354, respectively; P < .001). The technical skill domain had the highest utilization (98% of PBAs) and was the best predictor of cS and pS (rs : +0.96 and +0.66, respectively).
  
  
  CONCLUSIONS
  PBA is reliable and valid for assessing otolaryngology trainees' performance and progress at all levels. It is highly sensitive in identifying competent trainees. The tool is used in a formative and feedback capacity. The technical domain is the best predictor and should be given close attention.
  
  
  METHODS
  NA.
authors.affiliation: Department of Surgery and Cancer, Imperial College, London, United Kingdom.|Department of Otolaryngology, St. Mary's Hospital, Imperial College Healthcare NHS Trust, London, United Kingdom.
authors.forename: Zaid
authors.initials: Z
authors.lastname: Awad
authors.affiliation: School of Medicine, Imperial College, London, United Kingdom.
authors.forename: Lindsay
authors.initials: L
authors.lastname: Hayden
authors.affiliation: North Cumbria University Hospitals NHS Trust, Cumberland Infirmary, Carlisle, United Kingdom.
authors.forename: Andrew K
authors.initials: AK
authors.lastname: Robson
authors.affiliation: School of Medicine, Imperial College, London, United Kingdom.
authors.forename: Keerthini
authors.initials: K
authors.lastname: Muthuswamy
authors.affiliation: Department of Surgery and Cancer, Imperial College, London, United Kingdom.|Department of Otolaryngology, St. Mary's Hospital, Imperial College Healthcare NHS Trust, London, United Kingdom.
authors.forename: Neil S
authors.initials: NS
authors.lastname: Tolley
mesh_terms: D002983:Clinical Competence; D010036:Otolaryngology / Q000193:education*; D015203:Reproducibility of Results; D012189:Retrospective Studies
keywords: Training; assessment; postgraduate education