title: Feasibility, reliability and validity of an operative performance rating system for evaluating surgery residents.
abstract: BACKGROUND
  Resident evaluation traditionally involves global assessments including clinical performance, professional behavior, technical skill, and number of procedures performed. These evaluations lack objective assessment of operative skills. We describe an operative performance rating system (OPRS) designed to provide objective operative performance ratings using a sentinel procedure format.
  
  
  METHODS
  Ten-item procedure-specific rating instruments were developed. Items included technical skills, operative decision making, and general items. A 1 to 5 (5 = excellent) scale was used for evaluation. Six procedures had sufficient forms returned to allow evaluation. Inter-rater reliability was determined by having faculty evaluators view 2 videotaped operations.
  
  
  RESULTS
  Return rates for the Internet-based form were full-time faculty (92%), volunteer faculty (27%), and overall (67%). Reliability, (average interitem correlation), and total procedures evaluated were excisional biopsy, 0.90, (0.48), 77; open inguinal herniorraphy, 0.94, (0.62), 51; laparoscopic cholecystectomy, 0.95, (0.64), 75; small-bowel and colon resection, 0.92, (0.58), 30; parathyroidectomy, 0.70, (0.19), 30; and lumpectomy, 0.92, (0.51), 38. Years of training accounted for 25% to 57% of the variation in scores. Inter-rater variability was observed; however, the average rater agreement was reliable.
  
  
  CONCLUSIONS
  Internet-based management made obtaining the data feasible. The OPRS complements traditional evaluations by providing objective assessment of operative decision-making and technical skills. Interitem correlations indicate the average rating of items provides a reliable indicator of resident performance. The OPRS is useful in tracking resident development throughout postgraduate training and offers a structured means of certifying operative skills.
authors.affiliation: Department of Surgery, Southern Illinois University School of Medicine, Springfield 62794-9638, USA.
authors.forename: Jennine L
authors.initials: JL
authors.lastname: Larson
authors.forename: Reed G
authors.initials: RG
authors.lastname: Williams
authors.forename: Janet
authors.initials: J
authors.lastname: Ketchum
authors.forename: Margaret L
authors.initials: ML
authors.lastname: Boehler
authors.forename: Gary L
authors.initials: GL
authors.lastname: Dunnington
mesh_terms: D001706:Biopsy; D017081:Cholecystectomy, Laparoscopic; D002983:Clinical Competence; D013505:Digestive System Surgical Procedures; D004521:Educational Measurement / Q000379:methods*; D005240:Feasibility Studies; D013502:General Surgery / Q000193:education*; D006801:Humans; D007396:Internship and Residency; D015412:Mastectomy, Segmental; D016105:Parathyroidectomy; D015203:Reproducibility of Results