title: Crowd-sourced assessment of surgical skills in cricothyrotomy procedure.
abstract: BACKGROUND
  Objective assessment of surgical skills is resource intensive and requires valuable time of expert surgeons. The goal of this study was to assess the ability of a large group of laypersons using a crowd-sourcing tool to grade a surgical procedure (cricothyrotomy) performed on a simulator. The grading included an assessment of the entire procedure by completing an objective assessment of technical skills survey.
  
  
  METHODS
  Two groups of graders were recruited as follows: (1) Amazon Mechanical Turk users and (2) three expert surgeons from University of Washington Department of Otolaryngology. Graders were presented with a video of participants performing the procedure on the simulator and were asked to grade the video using the objective assessment of technical skills questions. Mechanical Turk users were paid $0.50 for each completed survey. It took 10 h to obtain all responses from 30 Mechanical Turk users for 26 training participants (26 videos/tasks), whereas it took 60 d for three expert surgeons to complete the same 26 tasks.
  
  
  RESULTS
  The assessment of surgical performance by a group (n = 30) of laypersons matched the assessment by a group (n = 3) of expert surgeons with a good level of agreement determined by Cronbach alpha coefficient = 0.83.
  
  
  CONCLUSIONS
  We found crowd sourcing was an efficient, accurate, and inexpensive method for skills assessment with a good level of agreement to experts' grading.
authors.affiliation: Department of Electrical Engineering, University of Washington, Seattle, Washington. Electronic address: navaa@u.washington.edu.
authors.forename: Nava
authors.initials: N
authors.lastname: Aghdasi
authors.affiliation: Department of Otolaryngology - Head and Neck Surgery, University of Washington, Seattle, Washington.
authors.forename: Randall
authors.initials: R
authors.lastname: Bly
authors.affiliation: M.D.C. School of Medicine, Stanford University, Stanford, California.
authors.forename: Lee W
authors.initials: LW
authors.lastname: White
authors.affiliation: Department of Electrical Engineering, University of Washington, Seattle, Washington.
authors.forename: Blake
authors.initials: B
authors.lastname: Hannaford
authors.affiliation: Department of Otolaryngology - Head and Neck Surgery, University of Washington, Seattle, Washington.
authors.forename: Kris
authors.initials: K
authors.lastname: Moe
authors.affiliation: Department of Urology, Seattle Children's Hospital Seattle, Washington.
authors.forename: Thomas S
authors.initials: TS
authors.lastname: Lendvay
mesh_terms: D002983:Clinical Competence / Q000592:standards*; D063045:Crowdsourcing; D006801:Humans; D013514:Surgical Procedures, Operative / Q000193:education / Q000592:standards*
keywords: Crowd sourcing; Emergency cricothyrotomy procedure; Surgical skill assessment