title: Use of an error-focused checklist to identify incompetence in lumbar puncture performances.
abstract: BACKGROUND
  Checklists are commonly used in the assessment of procedural competence. However, on most checklists, high scores are often unable to rule out incompetence as the commission of a few serious procedural errors typically results in only a minimal reduction in performance score. We hypothesised that checklists constructed based on procedural errors may be better at identifying incompetence.
  
  
  OBJECTIVE
  This study sought to compare the efficacy of an error-focused checklist and a conventionally constructed checklist in identifying procedural incompetence.
  
  
  METHODS
  We constructed a 15-item error-focused checklist for lumbar puncture (LP) based on input from 13 experts in four Canadian academic centres, using a modified Delphi approach, over three rounds of survey. Ratings of 18 video-recorded performances of LP on simulators using the error-focused tool were compared with ratings obtained using a published conventional 21-item checklist. Competence/incompetence decisions were based on global assessment. Diagnostic accuracy was estimated using the area under the curve (AUC) in receiver operating characteristic analyses.
  
  
  RESULTS
  The accuracy of the conventional checklist in identifying incompetence was low (AUC 0.11, 95% confidence interval [CI] 0.00-0.28) in comparison with that of the error-focused checklist (AUC 0.85, 95% CI 0.67-1.00). The internal consistency of the error-focused checklist was lower than that of the conventional checklist (α = 0.35 and α = 0.79, respectively). The inter-rater reliability of both tools was high (conventional checklist: intraclass correlation coefficient [ICC] 0.99, 95% CI 0.98-1.00; error-focused checklist: ICC 0.92, 95% CI 0.68-0.98).
  
  
  CONCLUSIONS
  Despite higher internal consistency and inter-rater reliability, the conventional checklist was less accurate at identifying procedural incompetence. For assessments in which it is important to identify procedural incompetence, we recommend the use of an error-focused checklist.
authors.affiliation: Department of Medicine, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.|W21C, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
authors.forename: Irene W Y
authors.initials: IW
authors.lastname: Ma
authors.affiliation: Department of Medicine, Faculty of Medicine, University of Ottawa, Ottawa, Ontario, Canada.
authors.forename: Debra
authors.initials: D
authors.lastname: Pugh
authors.affiliation: Department of Critical Care Medicine, Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
authors.forename: Briseida
authors.initials: B
authors.lastname: Mema
authors.affiliation: Department of Surgery, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
authors.forename: Mary E
authors.initials: ME
authors.lastname: Brindle
authors.affiliation: Department of Clinical Neurosciences, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
authors.forename: Lara
authors.initials: L
authors.lastname: Cooke
authors.affiliation: W21C, Faculty of Medicine, University of Calgary, Calgary, Alberta, Canada.
authors.forename: Julie N
authors.initials: JN
authors.lastname: Stromer
mesh_terms: D057189:Checklist; D002983:Clinical Competence; D006801:Humans; D019300:Medical Errors; D015588:Observer Variation; D013129:Spinal Puncture / Q000592:standards; D014741:Video Recording