title: Pilot testing of an assessment tool for competency in mastoidectomy.
abstract: OBJECTIVE
  To determine the feasibility, validity, and reliability of an evaluation tool for the assessment of competency in mastoid surgery. This study tests the hypothesis that residents of dissimilar training levels differ in their technical performance as measured by this tool.
  
  
  METHODS
  Cross-sectional validation study.
  
  
  METHODS
  Two or more faculty otolaryngologists evaluated each resident performing a cortical mastoidectomy on a cadaveric temporal bone. Performance was rated using global and checklist components of a mastoidectomy assessment tool.
  
  
  RESULTS
  Fifteen internal and external faculty members evaluated 23 residents over 3 years resulting in 118 evaluations. Construct validity was observed as scores increased with clinical training year. These differences were greater for more complex tasks. There was a percentage agreement of 78.5% between evaluator pairs in the determination of pass (score 3-5) versus fail (score 1-2) for the checklist instrument, and an agreement of 74.4% for the global instrument. Although agreement was lower for the exact score on a scale of 1 to 5, differences of 1 or less occurred in over 80% of evaluator pairs. Regression analysis confirmed faculty perception that sharpening of the posterior external auditory canal cortex and opening of the antrum from posterior to anterior are strong predictors of overall surgical performance.
  
  
  CONCLUSIONS
  Our results indicate that the tool we have developed is a feasible, valid, and reliable instrument for the assessment of competency in mastoidectomy. The instrument can be used to provide formative feedback and to identify procedural tasks for which additional training may be necessary.
authors.affiliation: Department Otolaryngology-Head and Neck Surgery, Johns Hopkins University, Baltimore, Maryland, USA.
authors.forename: Kulsoom
authors.initials: K
authors.lastname: Laeeq
authors.forename: Nasir I
authors.initials: NI
authors.lastname: Bhatti
authors.forename: John P
authors.initials: JP
authors.lastname: Carey
authors.forename: Charles C
authors.initials: CC
authors.lastname: Della Santina
authors.forename: Charles J
authors.initials: CJ
authors.lastname: Limb
authors.forename: John K
authors.initials: JK
authors.lastname: Niparko
authors.forename: Lloyd B
authors.initials: LB
authors.lastname: Minor
authors.forename: Howard W
authors.initials: HW
authors.lastname: Francis
mesh_terms: D002102:Cadaver; D002983:Clinical Competence / Q000592:standards*; D004650:Employee Performance Appraisal / Q000379:methods*; D006801:Humans; D007396:Internship and Residency / Q000592:standards*; D008416:Mastoid / Q000601:surgery*; D013506:Otologic Surgical Procedures / Q000193:education* / Q000592:standards*; D010865:Pilot Projects; D015203:Reproducibility of Results