title: Crowd-sourced assessment of technical skills: an adjunct to urology resident surgical simulation training.
abstract: BACKGROUND
  Crowdsourcing is the practice of obtaining services from a large group of people, typically an online community. Validated methods of evaluating surgical video are time-intensive, expensive, and involve participation of multiple expert surgeons. We sought to obtain valid performance scores of urologic trainees and faculty on a dry-laboratory robotic surgery task module by using crowdsourcing through a web-based grading tool called Crowd Sourced Assessment of Technical Skill (CSATS).
  
  
  METHODS
  IRB approval was granted to test the technical skills grading accuracy of Amazon.com Mechanical Turk™ crowd-workers compared to three expert faculty surgeon graders. The two groups assessed dry-laboratory robotic surgical suturing performances of three urology residents (PGY-2, -4, -5) and two faculty using three performance domains from the validated Global Evaluative Assessment of Robotic Skills assessment tool.
  
  
  RESULTS
  After an average of 2 hours 50 minutes, each of the five videos received 50 crowd-worker assessments. The inter-rater reliability (IRR) between the surgeons and crowd was 0.91 using Cronbach's alpha statistic (confidence intervals=0.20-0.92), indicating an agreement level between the two groups of "excellent." The crowds were able to discriminate the surgical level, and both the crowds and the expert faculty surgeon graders scored one senior trainee's performance above a faculty's performance.
  
  
  CONCLUSIONS
  Surgery-naive crowd-workers can rapidly assess varying levels of surgical skill accurately relative to a panel of faculty raters. The crowds provided rapid feedback and were inexpensive. CSATS may be a valuable adjunct to surgical simulation training as requirements for more granular and iterative performance tracking of trainees become mandated and commonplace.
authors.affiliation: 1 University of Washington School of Medicine , Seattle, Washington.
authors.forename: Daniel
authors.initials: D
authors.lastname: Holst
authors.forename: Timothy M
authors.initials: TM
authors.lastname: Kowalewski
authors.forename: Lee W
authors.initials: LW
authors.lastname: White
authors.forename: Timothy C
authors.initials: TC
authors.lastname: Brand
authors.forename: Jonathan D
authors.initials: JD
authors.lastname: Harper
authors.forename: Mathew D
authors.initials: MD
authors.lastname: Sorenson
authors.forename: Sarah
authors.initials: S
authors.lastname: Kirsch
authors.forename: Thomas S
authors.initials: TS
authors.lastname: Lendvay
mesh_terms: D002983:Clinical Competence; D063045:Crowdsourcing / Q000379:methods; D004521:Educational Measurement / Q000379:methods; D006801:Humans; D007396:Internship and Residency; D010820:Physicians; D015203:Reproducibility of Results; D065287:Robotic Surgical Procedures / Q000193:education*; D000066908:Simulation Training; D013536:Suture Techniques / Q000193:education*; D013520:Urologic Surgical Procedures / Q000193:education; D014572:Urology / Q000193:education*; D014741:Video Recording