title: Diagnosing technical competence in six bedside procedures: comparing checklists and a global rating scale in the assessment of resident performance.
abstract: OBJECTIVE
  To compare procedure-specific checklists and a global rating scale in assessing technical competence.
  
  
  METHODS
  Two trained raters used procedure-specific checklists and a global rating scale to independently evaluate 218 video-recorded performances of six bedside procedures of varying complexity for technical competence. The procedures were completed by 47 residents participating in a formative simulation-based objective structured clinical examination at the University of Calgary in 2011. Pass/fail (competent/not competent) decisions were based on an overall global assessment item on the global rating scale. Raters provided written comments on performances they deemed not competent. Checklist minimum passing levels were set using traditional standard-setting methods.
  
  
  RESULTS
  For each procedure, the global rating scale demonstrated higher internal reliability and lower interrater reliability than the checklist. However, interrater reliability was almost perfect for decisions on competence using the overall global assessment (Kappa range: 0.84-1.00). Clinically significant procedural errors were most often cited as reasons for ratings of not competent. Using checklist scores to diagnose competence demonstrated acceptable discrimination: The area under the curve ranged from 0.84 (95% CI 0.72-0.97) to 0.93 (95% CI 0.82-1.00). Checklist minimum passing levels demonstrated high sensitivity but low specificity for diagnosing competence.
  
  
  CONCLUSIONS
  Assessment using a global rating scale may be superior to assessment using a checklist for evaluation of technical competence. Traditional standard-setting methods may establish checklist cut scores with too-low specificity: High checklist scores did not rule out incompetence. The role of clinically significant errors in determining procedural competence should be further evaluated.
authors.affiliation: A. Walzak is clinical instructor, Department of Medicine, University of British Columbia, Victoria, British Columbia, Canada. M. Bacchus is associate professor, Department of Medicine, University of Calgary, Calgary, Alberta, Canada. J.P. Schaefer is clinical professor, Department of Medicine, University of Calgary, Calgary, Alberta, Canada. K. Zarnke is associate professor, Department of Medicine, University of Calgary, Calgary, Alberta, Canada. J. Glow is internal medicine residency program administrator, University of Calgary, Calgary, Alberta, Canada. C. Brass is internal medicine residency program assistant, University of Calgary, Calgary, Alberta, Canada. K. McLaughlin is associate professor, Department of Medicine, University of Calgary, Calgary, Alberta, Canada. I.W.Y. Ma is associate professor, Department of Medicine, University of Calgary, Calgary, Alberta, Canada.
authors.forename: Alison
authors.initials: A
authors.lastname: Walzak
authors.forename: Maria
authors.initials: M
authors.lastname: Bacchus
authors.forename: Jeffrey P
authors.initials: JP
authors.lastname: Schaefer
authors.forename: Kelly
authors.initials: K
authors.lastname: Zarnke
authors.forename: Jennifer
authors.initials: J
authors.lastname: Glow
authors.forename: Charlene
authors.initials: C
authors.lastname: Brass
authors.forename: Kevin
authors.initials: K
authors.lastname: McLaughlin
authors.forename: Irene W Y
authors.initials: IW
authors.lastname: Ma
mesh_terms: D000328:Adult; D000416:Alberta; D057189:Checklist; D002983:Clinical Competence; D004503:Education, Medical, Graduate; D004521:Educational Measurement / Q000379:methods*; D004650:Employee Performance Appraisal / Q000379:methods*; D005260:Female; D006801:Humans; D007388:Internal Medicine / Q000193:education*; D007396:Internship and Residency; D008297:Male; D019300:Medical Errors / Q000706:statistics & numerical data; D019095:Point-of-Care Systems / Q000592:standards*; D015203:Reproducibility of Results; D014743:Videotape Recording