title: Crowd-Sourced Assessment of Technical Skill: A Valid Method for Discriminating Basic Robotic Surgery Skills.
abstract: BACKGROUND
  A surgeon's skill in the operating room has been shown to correlate with a patient's clinical outcome. The prompt accurate assessment of surgical skill remains a challenge, in part, because expert faculty reviewers are often unavailable. By harnessing the power of large readily available crowds through the Internet, rapid, accurate, and low-cost assessments may be achieved. We hypothesized that assessments provided by crowd workers highly correlate with expert surgeons' assessments.
  
  
  METHODS
  A group of 49 surgeons from two hospitals performed two dry-laboratory robotic surgical skill assessment tasks. The performance of these tasks was video recorded and posted online for evaluation using Amazon Mechanical Turk. The surgical tasks in each video were graded by (n=30) varying crowd workers and (n=3) experts using a modified global evaluative assessment of Robotic Skills (GEARS) grading tool, and the mean scores were compared using Cronbach's alpha statistic.
  
  
  RESULTS
  GEARS evaluations from the crowd were obtained for each video and task and compared with the GEARS ratings from the expert surgeons. The crowd-based performance scores agreed with the performance assessments by experts with a Cronbach's alpha of 0.84 and 0.92 for the two tasks, respectively.
  
  
  CONCLUSIONS
  The assessment of surgical skill by crowd workers resulted in a high degree of agreement with the scores provided by expert surgeons in the evaluation of basic robotic surgical dry-laboratory tasks. Crowd responses cost less and were much faster to acquire. This study provides evidence that crowds may provide an adjunctive method for rapidly providing feedback of skills to training and practicing surgeons.
authors.affiliation: 1 School of Medicine, Stanford University, Palo Alto, California. (At time of data collection and analysis: Department of Bioengineering, University of Washington , Seattle, Washington.).
authors.forename: Lee W
authors.initials: LW
authors.lastname: White
authors.affiliation: 2 Department of Mechanical Engineering, University of Minnesota , Minneapolis, Minnesota.
authors.forename: Timothy M
authors.initials: TM
authors.lastname: Kowalewski
authors.affiliation: 2 Department of Mechanical Engineering, University of Minnesota , Minneapolis, Minnesota.
authors.forename: Rodney Lee
authors.initials: RL
authors.lastname: Dockter
authors.affiliation: 3 Department of Biostatistics, University of Washington , Seattle, Washington.
authors.forename: Bryan
authors.initials: B
authors.lastname: Comstock
authors.affiliation: 4 Department of Electrical Engineering, University of Washington , Seattle, Washington.
authors.forename: Blake
authors.initials: B
authors.lastname: Hannaford
authors.affiliation: 5 Department of Urology, Seattle Children's Hospital, University of Washington Medical Center, University of Washington , Seattle, Washington.
authors.forename: Thomas S
authors.initials: TS
authors.lastname: Lendvay
mesh_terms: D000328:Adult; D002983:Clinical Competence; D063045:Crowdsourcing; D005260:Female; D013502:General Surgery / Q000193:education / Q000592:standards; D006176:Gynecology / Q000193:education / Q000592:standards; D006801:Humans; D020407:Internet; D007396:Internship and Residency; D008297:Male; D008875:Middle Aged; D009774:Obstetrics / Q000193:education / Q000592:standards; D015203:Reproducibility of Results; D065287:Robotic Surgical Procedures / Q000193:education / Q000592:standards*; D014572:Urology / Q000193:education / Q000592:standards; D014741:Video Recording