Pilot testing of an assessment tool for competency in mastoidectomy.
OBJECTIVE
To determine the feasibility, validity, and reliability of an evaluation tool for the assessment of competency in mastoid surgery. This study tests the hypothesis that residents of dissimilar training levels differ in their technical performance as measured by this tool.


METHODS
Cross-sectional validation study.


METHODS
Two or more faculty otolaryngologists evaluated each resident performing a cortical mastoidectomy on a cadaveric temporal bone. Performance was rated using global and checklist components of a mastoidectomy assessment tool.


RESULTS
Fifteen internal and external faculty members evaluated 23 residents over 3 years resulting in 118 evaluations. Construct validity was observed as scores increased with clinical training year. These differences were greater for more complex tasks. There was a percentage agreement of 78.5% between evaluator pairs in the determination of pass (score 3-5) versus fail (score 1-2) for the checklist instrument, and an agreement of 74.4% for the global instrument. Although agreement was lower for the exact score on a scale of 1 to 5, differences of 1 or less occurred in over 80% of evaluator pairs. Regression analysis confirmed faculty perception that sharpening of the posterior external auditory canal cortex and opening of the antrum from posterior to anterior are strong predictors of overall surgical performance.


CONCLUSIONS
Our results indicate that the tool we have developed is a feasible, valid, and reliable instrument for the assessment of competency in mastoidectomy. The instrument can be used to provide formative feedback and to identify procedural tasks for which additional training may be necessary.
Laeeq
Kulsoom
K
Department Otolaryngology-Head and Neck Surgery, Johns Hopkins University, Baltimore, Maryland, USA.
Bhatti
Nasir I
NI
Carey
John P
JP
Della Santina
Charles C
CC
Limb
Charles J
CJ
Niparko
John K
JK
Minor
Lloyd B
LB
Francis
Howard W
HW
D002102:Cadaver; D002983:Clinical Competence / Q000592:standards*; D004650:Employee Performance Appraisal / Q000379:methods*; D006801:Humans; D007396:Internship and Residency / Q000592:standards*; D008416:Mastoid / Q000601:surgery*; D013506:Otologic Surgical Procedures / Q000193:education* / Q000592:standards*; D010865:Pilot Projects; D015203:Reproducibility of Results