Continuous, data-rich appraisal of surgical trainees' operative abilities: a novel approach for measuring performance and providing feedback.
BACKGROUND
We developed a convenient mechanism, Surgical Training and Assessment Tool (STAT), to accomplish detailed, continuous analysis of surgical trainees' operative abilities, and a simple method, Quality Based Surgical Training (QBST) for implementing it.


METHODS
Using a web-accessed computer program, attending physicians and trainees independently assessed the trainee's operative performance after every operative (training) case. Global attributes of surgical knowledge, skill, and independence were assessed as well as the key technical maneuvers of each operation. A system of hierarchical, expandable menus specific to each of hundreds of different surgical procedures allowed the assessments to be made as detailed or as general as the users felt were necessary. In addition, freehand, unscripted commentary was recorded via an optional "remarks" box feature. Finally, an independently chosen, "overall" grade scaled F through A+ concluded each assessment.


RESULTS
Over a 31 month period, 72 different users (52 trainees, 20 attending physicians) submitted 3849 performance assessments on 2424 cases, including 132 different case types and amassing 68,260 distinct data points. The mean number of data points per trainee was 1313; the median time spent per assessment was 60 seconds. Graphic displays allowed formative review of individual cases in real time, and summative review of long term trends. Appraisals of knowledge, skill, and independence were strongly correlated with and independently predictive of the overall competency grade (model r(2) = 0.68; test of predictive significance p < 0.001 for each rating). Trainee and attending physician scores were highly correlated (> 0.7) with one another.


CONCLUSIONS
QBST/STAT achieves detailed, continuous analysis of surgical trainees' operative abilities, and facilitates timely, specific, and thorough feedback regarding their performance in theater. QBST/STAT promotes trainee self-reflection and generation of continuous, transparent, iterative training goals.
Roach
Paul B
PB
Department of Surgery, Naval Medical Center Portsmouth, Portsmouth, Virginia.
Roggin
Kevin K
KK
Selkov
Gene
G
Posner
Mitchell C
MC
Silverstein
Jonathan C
JC
D002983:Clinical Competence; D004503:Education, Medical, Graduate / Q000379:methods*; D004521:Educational Measurement / Q000379:methods*; D005246:Feedback; D005260:Female; D006801:Humans; D007396:Internship and Residency; D008297:Male; D016347:Medical Records Systems, Computerized; D008505:Medical Staff, Hospital; D011787:Quality of Health Care; D013043:Specialties, Surgical / Q000193:education*; D013514:Surgical Procedures, Operative / Q000193:education*; D013647:Task Performance and Analysis; D014481:United States