title: A template for reliable assessment of resident operative performance: assessment intervals, numbers of cases and raters.
abstract: BACKGROUND
  Operative performance rating (OPR) instruments have been developed to assess operative performance (OP). To guide program implementation, this study determined: 1) Appropriate intervals for OP progress decisions, 2) Number of OPRs and raters required per interval to achieve reproducible results.
  
  
  METHODS
  21 surgeons rated 897 OPs (3 procedures) by 36 residents. Six-month PGY intervals were compared to determine length of stable operative performance intervals. Variance component analyses established rating factor importance. Generalizability analyses and decision studies determined number of OPRs required for reproducible OP decisions (reliabilities = 0.80).
  
  
  RESULTS
  Resident OPRs are stable across single PGY years. 2.3 OPRs/resident/month provided a dependable basis for annual or semi-annual resident OP decisions. Results were similar for all procedures and training years. Rater idiosyncrasies accounted for most score variation (63% when interaction effects involving rater idiosyncrasies were included). Resident ability was the next most important source of variation (12%). Procedure was a less important source (5%).
  
  
  CONCLUSIONS
  Annual resident OP decisions are supported. 2.3 OPRs per month provide a dependable basis for judging resident OP. These numbers are sufficient regardless of training year or procedure mix though efforts should be made to balance procedure mix. Multiple raters should rate each resident to control for rater idiosyncrasies.
authors.affiliation: Department of Surgery, Southern Illinois University School of Medicine, Springfield, IL, USA. rwilliams@siumed.edu
authors.forename: Reed G
authors.initials: RG
authors.lastname: Williams
authors.forename: Steven
authors.initials: S
authors.lastname: Verhulst
authors.forename: Jerry A
authors.initials: JA
authors.lastname: Colliver
authors.forename: Hilary
authors.initials: H
authors.lastname: Sanfey
authors.forename: Xiaodong
authors.initials: X
authors.lastname: Chen
authors.forename: Gary L
authors.initials: GL
authors.lastname: Dunnington
mesh_terms: D000046:Academic Medical Centers; D002983:Clinical Competence / Q000592:standards; D004521:Educational Measurement / Q000379:methods; D013502:General Surgery / Q000193:education* / Q000592:standards*; D006801:Humans; D007087:Illinois; D007396:Internship and Residency / Q000592:standards*; D010820:Physicians / Q000592:standards; D012577:Schools, Medical