title: Objective structured assessment of technical skills and checklist scales reliability compared for high stakes assessments.
abstract: BACKGROUND
  The establishment of assessment reliability at the level of the individual trainee is an important attribute of assessment methodologies, particularly for doctors who have been failed. This issue is of particular importance for the process of competence assessment in the USA, UK, Australia and New Zealand.
  
  
  METHODS
  We use data from 19 applicants for higher surgical training in 2008 at the Royal College of Surgeons in Ireland to compare: (i) the objective structured assessment of technical skills (OSATS) method; and (ii) a procedure-specific checklist to assess surgical technical skills in the excision of a sebaceous cyst task by two experienced senior surgeons.
  
  
  RESULTS
  The overall interrater reliability (IRR) of the OSATS assessment as determined by a correlation coefficient was 0.507 (P < 0.03) and 0.67 with coefficient alpha, considerably below the accepted 0.8 level of IRR. The checklist's overall IRR was 0.89. Individually, only five (26%) of the OSATS assessments reached the 0.8 level of IRR in contrast to 18 (95%) of the checklist assessments.
  
  
  CONCLUSIONS
  We propose binary procedure-based assessment checklists as more reliable assessment instruments with more robust reproducibility.
authors.affiliation: School of Medicine, University College Cork, Cork, Ireland; National Surgical Training Centre, Royal College of Surgeons in Ireland, Dublin, Ireland.
authors.forename: Anthony G
authors.initials: AG
authors.lastname: Gallagher
authors.forename: Gerald C
authors.initials: GC
authors.lastname: O'Sullivan
authors.forename: Gerald
authors.initials: G
authors.lastname: Leonard
authors.forename: Brendan P
authors.initials: BP
authors.lastname: Bunting
authors.forename: Kieran J
authors.initials: KJ
authors.lastname: McGlade
mesh_terms: D000328:Adult; D057189:Checklist; D002983:Clinical Competence; D004503:Education, Medical, Graduate; D004814:Epidermal Cyst / Q000601:surgery*; D005260:Female; D013502:General Surgery / Q000193:education*; D006801:Humans; D007494:Ireland; D008297:Male; D015203:Reproducibility of Results; D012570:School Admission Criteria