The development of an independent rater system to assess residents' competence in invasive procedures.
OBJECTIVE
To design an independent rater (IR) direct observation system to monitor invasive procedures performed by residents in the hospital setting.


METHODS
The authors recruited, trained, and tested nonphysicians to become IRs for an Agency for Healthcare Research and Quality-funded study evaluating the impact of partial task simulation training of ultrasound-guided central venous catheter (CVC) insertion on skills transfer at a major academic medical center. IR applicants completed four hours of training: a two-hour didactic session and a two-hour testing session, including observation of 5 of 10 choreographed CVC insertion videotapes and completion of a 50-data-point procedural checklist. Eligibility to be hired as an IR included timing the procedure accurately, detecting technical errors and complications, and completing the procedural checklist accurately.


RESULTS
Thirty-eight IR trainees completed the training module and videotape examinations. Twenty-seven (71%) trainees met criteria to be hired IRs by accurately assessing the duration of the procedure to within one minute, validating the checklist to within 95% accuracy, and detecting technical errors/complications to within a 3% margin of error. The authors found no association between educational level and hired status, and all 13 IRs assessed after the study had maintained their skills.


CONCLUSIONS
Recent innovations in procedural training with partial task simulation trainers necessitate developing methods to measure skills transfer from the simulator to the clinical setting. This description of a nonphysician IR direct observation system for CVC insertion offers a feasible tool that may be generalized to monitoring other invasive procedures.
Evans
Leigh V
LV
Yale University School of Medicine, Section of Emergency Medicine, New Haven, Connecticut 06519, USA. leigh.evans@yale.edu
Morse
James L
JL
Hamann
Cara J
CJ
Osborne
Michael
M
Lin
Zhenqiu
Z
D'Onofrio
Gail
G
D000328:Adult; D000488:Allied Health Personnel; D002405:Catheterization, Central Venous / Q000592:standards*; D002983:Clinical Competence / Q000592:standards*; D004503:Education, Medical, Graduate / Q000592:standards*; D004521:Educational Measurement / Q000379:methods*; D005260:Female; D006801:Humans; D007396:Internship and Residency; D007600:Judgment; D008297:Male; D015203:Reproducibility of Results; D018084:Ultrasonography, Interventional / Q000592:standards*