title: C-SATS: Assessing Surgical Skills Among Urology Residency Applicants.
abstract: We hypothesized that surgical skills assessment could aid in the selection process of medical student applicants to a surgical program. Recently, crowdsourcing has been shown to provide an accurate assessment of surgical skills at all levels of training. We compared expert and crowd assessment of surgical tasks performed by resident applicants during their interview day at the urology program at the University of California, Irvine.
  
  
  
  Twenty-five resident interviewees performed four tasks: open square knot tying, laparoscopic peg transfer, robotic suturing, and skill task 8 on the LAP Mentor™ (Simbionix Ltd., Lod, Israel). Faculty experts and crowd workers (Crowd-Sourced Assessment of Technical Skills [C-SATS], Seattle, WA) assessed recorded performances using the Objective Structured Assessment of Technical Skills (OSATS), Global Evaluative Assessment of Robotic Skills (GEARS), and the Global Operative Assessment of Laparoscopic Skills (GOALS) validated assessment tools.
  
  
  
  Overall, 3938 crowd assessments were obtained for the four tasks in less than 3.5 hours, whereas the average time to receive 150 expert assessments was 22 days. Inter-rater agreement between expert and crowd assessment scores was 0.62 for open knot tying, 0.92 for laparoscopic peg transfer, and 0.86 for robotic suturing. Agreement between applicant rank on skill task 8 on the LAP Mentor assessment and crowd assessment was 0.32. The crowd match rank based solely on skills performance did not compare well with the final faculty match rank list (0.46); however, none of the bottom five crowd-rated applicants appeared in the top five expert-rated applicants and none of the top five crowd-rated applicants appeared in the bottom five expert-rated applicants.
  
  
  
  Crowd-source assessment of resident applicant surgical skills has good inter-rater agreement with expert physician raters but not with a computer-based objective motion metrics software assessment. Overall applicant rank was affected to some degree by the crowd performance rating.
authors.affiliation: 1 Department of Urology, University of California , Irvine, Orange, California.
authors.forename: Simone L
authors.initials: SL
authors.lastname: Vernez
authors.affiliation: 1 Department of Urology, University of California , Irvine, Orange, California.
authors.forename: Victor
authors.initials: V
authors.lastname: Huynh
authors.affiliation: 2 Hematology-Oncology Division, Department of Medicine, University of California , Irvine, Orange, California.
authors.forename: Kathryn
authors.initials: K
authors.lastname: Osann
authors.affiliation: 1 Department of Urology, University of California , Irvine, Orange, California.
authors.forename: Zhamshid
authors.initials: Z
authors.lastname: Okhunov
authors.affiliation: 1 Department of Urology, University of California , Irvine, Orange, California.
authors.forename: Jaime
authors.initials: J
authors.lastname: Landman
authors.affiliation: 1 Department of Urology, University of California , Irvine, Orange, California.
authors.forename: Ralph V
authors.initials: RV
authors.lastname: Clayman
mesh_terms: D002983:Clinical Competence; D063045:Crowdsourcing; D006801:Humans; D007396:Internship and Residency; D010535:Laparoscopy; D010560:Personnel Selection; D010820:Physicians; D065287:Robotic Surgical Procedures; D012984:Software; D013337:Students, Medical; D013536:Suture Techniques; D013647:Task Performance and Analysis; D014572:Urology / Q000193:education*
keywords: clinical competence; crowdsourcing; education; interviews; residency; validation studies