title: A controlled study to determine measurement conditions necessary for a reliable and valid operative performance assessment: a controlled prospective observational study.
abstract: OBJECTIVE
  This study evaluated operative performance rating (OPR) characteristics and measurement conditions necessary for reliable and valid operative performance (OP) assessment.
  
  
  BACKGROUND
  Operative performance is a signature surgical-practice characteristic that is not measured systematically and specifically during residency training.
  
  
  METHODS
  Expert surgeon raters from multiple institutions, blinded to resident characteristics, independently evaluated 8 open and laparoscopic OP recordings immediately after observation.
  
  
  RESULTS
  A plurality of raters agreed on operative performance ratings (OPRs) for all performances. Using 10 judges adjusted for rater idiosyncrasies. Interrater agreement was similar for procedure-specific and general items. Higher post graduate year (PGY) residents received higher OPRs. Supervising-surgeon ratings averaged 0.51 points (1.2 standard deviations) above expert ratings for the same performances.
  
  
  CONCLUSIONS
  OPRs have measurement properties (reliability, validity) similar to those of other well-developed performance assessments (Mini-CEX [clinical evaluation exercise], standardized patient examinations) when ratings occur immediately after observation. OPRs by blinded expert judges reflect the level of resident training and are practically significant differences as the average rating for PGY 4 residents corresponded to a "Good" performance whereas those for PGY 5 residents corresponded to a "Very Good" performance. Supervising surgeon ratings are higher than expert judge ratings reflecting the effect of interpersonal factors on supervising surgeon ratings. Use of local and national norms for interpretation of OPRs would adjust for these interpersonal factors. The OPR system provides a practical means for measuring operative performance, which is a signature characteristic of surgical practice.
authors: Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL 62794, USA. rwilliams@siumed.edu
authors: Reed G
authors: RG
authors: Williams
authors: Hilary
authors: H
authors: Sanfey
authors: Xiaodong Phoenix
authors: XP
authors: Chen
authors: Gary L
authors: GL
authors: Dunnington
mesh_terms: D017081:Cholecystectomy, Laparoscopic / Q000592:standards; D002983:Clinical Competence; D003162:Competency-Based Education / Q000592:standards; D004521:Educational Measurement / Q000379:methods; D013502:General Surgery / Q000193:education*; D006801:Humans; D007396:Internship and Residency; D011446:Prospective Studies; D011594:Psychometrics; D015203:Reproducibility of Results; D013514:Surgical Procedures, Operative / Q000592:standards; D013647:Task Performance and Analysis