title: Assessment of resident surgical skills: is testing feasible?
abstract: OBJECTIVE
  We have previously shown that in a single residency program objective structured assessment of technical skills (OSATS) is a reliable and valid method of assessing surgical competency. Our goal was to establish feasibility of this evaluation instrument when administered at multiple residency programs throughout the US, and assess the impact of a laboratory-based surgical curriculum on results.
  
  
  METHODS
  An OSATS exam was administered to 116 residents from 5 residency programs. One of the residency programs had participated in a comprehensive surgical curriculum over a 4-year period of time. The exam consisted of 3 open and 3 laparoscopic tasks. Residents were graded by both a blinded and unblinded examiner with task-specific checklist, global rating scale, pass/fail assessment, and tasks were timed. All tasks were performed on life-like models.
  
  
  RESULTS
  Examinations were successfully completed at all sites. Each exam required 90 minutes of resident time. Reliability indices calculated with Cronbach's alpha were .97 for overall global rating and .95 for checklists. Interrater reliability between blinded and unblinded examiners ranged from .71 to .97 for individual tasks and was .95 overall. Assessment of construct validity (the ability to distinguish among residency levels) found significant differences among the residents for both blinded and unblinded examiners for all evaluation outcomes except time. For the test overall, the global rating scale showed significant differences among all 4 residency levels. The checklist showed significant differences at three levels (PGY3-4 >PGY2 >PGY1). Approximate cost for replaceable items was 40 US dollars to 150 per resident depending on which tasks were chosen. Comparison of scores between residents who received a laboratory-based curriculum and those who did not revealed significantly higher scores and shorter time to complete tasks for the group who received additional training.
  
  
  CONCLUSIONS
  Large-scale testing has confirmed that OSATS is an objective, reliable, and valid method to assess surgical skills, and can easily be administered in most residency programs. A laboratory-based surgical curriculum improved test results and reduced time to complete tasks.
authors.affiliation: Division of Gynecologic Oncology, University of Washington School of Medicine, Seattle, WA 98195, USA. bgoff@u.washington.edu
authors.forename: Barbara
authors.initials: B
authors.lastname: Goff
authors.forename: Lynn
authors.initials: L
authors.lastname: Mandel
authors.forename: Gretchen
authors.initials: G
authors.lastname: Lentz
authors.forename: Amy
authors.initials: A
authors.lastname: Vanblaricom
authors.forename: Anne-Marie Amies
authors.initials: AM
authors.lastname: Oelschlager
authors.forename: David
authors.initials: D
authors.lastname: Lee
authors.forename: Andrew
authors.initials: A
authors.lastname: Galakatos
authors.forename: Matthew
authors.initials: M
authors.lastname: Davies
authors.forename: Peter
authors.initials: P
authors.lastname: Nielsen
mesh_terms: D002983:Clinical Competence; D003479:Curriculum; D004503:Education, Medical, Graduate; D004521:Educational Measurement; D005260:Female; D013502:General Surgery / Q000193:education*; D006801:Humans; D007396:Internship and Residency; D008297:Male; D011336:Probability; D015203:Reproducibility of Results; D014481:United States