title: Crowd-sourced assessment of technical skills: an opportunity for improvement in the assessment of laparoscopic surgical skills.
abstract: BACKGROUND
  Objective, unbiased assessment of surgical skills remains a challenge in surgical education. We sought to evaluate the feasibility and reliability of Crowd-Sourced Assessment of Technical Skills.
  
  
  METHODS
  Seven volunteer general surgery interns were given time for training and then testing, on laparoscopic peg transfer, precision cutting, and intracorporeal knot-tying. Six faculty experts (FEs) and 203 Amazon.com Mechanical Turk crowd workers (CWs) evaluated 21 deidentified video clips using the Global Objective Assessment of Laparoscopic Skills validated rating instrument.
  
  
  RESULTS
  Within 19 hours and 15 minutes we received 662 eligible ratings from 203 CWs and 126 ratings from 6 FEs over 10 days. FE video ratings were of borderline internal consistency (Krippendorff's alpha = .55). FE ratings were highly correlated with CW ratings (Pearson's correlation coefficient = .78, P < .001).
  
  
  CONCLUSIONS
  We propose the use of Crowd-Sourced Assessment of Technical Skills as a reliable, basic tool to standardize the evaluation of technical skills in general surgery.
authors.affiliation: Virginia Mason Medical Center, Department of General Surgery, Mailstop H8-GME, 1100 9th Ave., Seattle, WA, 98101, USA. Electronic address: Shanley.Deal@virginiamason.org.
authors.forename: Shanley B
authors.initials: SB
authors.lastname: Deal
authors.affiliation: Department of Urology, University of Washington, Seattle, WA, USA.
authors.forename: Thomas S
authors.initials: TS
authors.lastname: Lendvay
authors.affiliation: Madigan Army Medical Center, Department of General Surgery, Tacoma, WA, USA.
authors.forename: Mohamad I
authors.initials: MI
authors.lastname: Haque
authors.affiliation: Madigan Army Medical Center, Department of General Surgery, Tacoma, WA, USA.
authors.forename: Timothy
authors.initials: T
authors.lastname: Brand
authors.affiliation: Department of Urology, University of Washington, Seattle, WA, USA.
authors.forename: Bryan
authors.initials: B
authors.lastname: Comstock
authors.affiliation: Department of Urology, University of Washington, Seattle, WA, USA.
authors.forename: Justin
authors.initials: J
authors.lastname: Warren
authors.affiliation: Virginia Mason Medical Center, Department of General Surgery, Mailstop H8-GME, 1100 9th Ave., Seattle, WA, 98101, USA.
authors.forename: Adnan
authors.initials: A
authors.lastname: Alseidi
mesh_terms: D002983:Clinical Competence; D063045:Crowdsourcing; D005240:Feasibility Studies; D013502:General Surgery / Q000193:education*; D006801:Humans; D007396:Internship and Residency; D010535:Laparoscopy / Q000193:education*; D010865:Pilot Projects; D015203:Reproducibility of Results; D013536:Suture Techniques / Q000193:education; D014741:Video Recording
keywords: Crowd sourced data; Psychomotor skills; Surgical skills assessment; Surgical skills education