title: Crowd-Sourced Assessment of Technical Skills: Differentiating Animate Surgical Skill Through the Wisdom of Crowds.
abstract: BACKGROUND
  Objective quantification of surgical skill is imperative as we enter a healthcare environment of quality improvement and performance-based reimbursement. The gold standard tools are infrequently used due to time-intensiveness, cost inefficiency, and lack of standard practices. We hypothesized that valid performance scores of surgical skill can be obtained through crowdsourcing.
  
  
  METHODS
  Twelve surgeons of varying robotic surgical experience performed live porcine robot-assisted urinary bladder closures. Blinded video-recorded performances were scored by expert surgeon graders and by Amazon's Mechanical Turk crowdsourcing crowd workers using the Global Evaluative Assessment of Robotic Skills tool assessing five technical skills domains. Seven expert graders and 50 unique Mechanical Turkers (each paid $0.75/survey) evaluated each video. Global assessment scores were analyzed for correlation and agreement.
  
  
  RESULTS
  Six hundred Mechanical Turkers completed the surveys in less than 5 hours, while seven surgeon graders took 14 days. The duration of video clips ranged from 2 to 11 minutes. The correlation coefficient between the Turkers' and expert graders' scores was 0.95 and Cronbach's Alpha was 0.93. Inter-rater reliability among the surgeon graders was 0.89.
  
  
  CONCLUSIONS
  Crowdsourcing surgical skills assessment yielded rapid inexpensive agreement with global performance scores given by expert surgeon graders. The crowdsourcing method may provide surgical educators and medical institutions with a boundless number of procedural skills assessors to efficiently quantify technical skills for use in trainee advancement and hospital quality improvement.
authors: 1 University of Washington School of Medicine , Seattle, Washington.
authors: Daniel
authors: D
authors: Holst
authors: 2 Department of Mechanical Engineering, University of Minnesota , Minneapolis, Minnesota.
authors: Timothy M
authors: TM
authors: Kowalewski
authors: 3 Stanford University School of Medicine , Palo Alto, California.
authors: Lee W
authors: LW
authors: White
authors: 4 Department of Urology, Madigan Army Medical Center , Tacoma.
authors: Timothy C
authors: TC
authors: Brand
authors: 5 Department of Urology, University of Washington , Seattle, Washington.
authors: Jonathan D
authors: JD
authors: Harper
authors: 5 Department of Urology, University of Washington , Seattle, Washington.
authors: Mathew D
authors: MD
authors: Sorensen
authors: 6 Florida Hospital Nicholson Center , Orlando, Florida.
authors: Mireille
authors: M
authors: Truong
authors: 6 Florida Hospital Nicholson Center , Orlando, Florida.
authors: Khara
authors: K
authors: Simpson
authors: 6 Florida Hospital Nicholson Center , Orlando, Florida.
authors: Alyssa
authors: A
authors: Tanaka
authors: 6 Florida Hospital Nicholson Center , Orlando, Florida.
authors: Roger
authors: R
authors: Smith
authors: 5 Department of Urology, University of Washington , Seattle, Washington.
authors: Thomas S
authors: TS
authors: Lendvay
mesh_terms: D000818:Animals; D002983:Clinical Competence / Q000592:standards*; D063045:Crowdsourcing / Q000379:methods*; D006801:Humans; D010535:Laparoscopy / Q000379:methods*; D015203:Reproducibility of Results; D065287:Robotic Surgical Procedures / Q000379:methods*; D011795:Surveys and Questionnaires; D013552:Swine; D001743:Urinary Bladder / Q000601:surgery*; D013520:Urologic Surgical Procedures / Q000379:methods*; D014741:Video Recording