title: Crowdsourcing Assessment of Surgeon Dissection of Renal Artery and Vein During Robotic Partial Nephrectomy: A Novel Approach for Quantitative Assessment of Surgical Performance.
abstract: BACKGROUND
  We sought to describe a methodology of crowdsourcing for obtaining quantitative performance ratings of surgeons performing renal artery and vein dissection of robotic partial nephrectomy (RPN). We sought to compare assessment of technical performance obtained from the crowdsourcers with that of surgical content experts (CE). Our hypothesis is that the crowd can score performances of renal hilar dissection comparably to surgical CE using the Global Evaluative Assessment of Robotic Skills (GEARS).
  
  
  METHODS
  A group of resident and attending robotic surgeons submitted a total of 14 video clips of RPN during hilar dissection. These videos were rated by both crowd and CE for technical skills performance using GEARS. A minimum of 3 CE and 30 Amazon Mechanical Turk crowdworkers evaluated each video with the GEARS scale.
  
  
  RESULTS
  Within 13 days, we received ratings of all videos from all CE, and within 11.5 hours, we received 548 GEARS ratings from crowdworkers. Even though CE were exposed to a training module, internal consistency across videos of CE GEARS ratings remained low (ICC = 0.38). Despite this, we found that crowdworker GEARS ratings of videos were highly correlated with CE ratings at both the video level (R = 0.82, p < 0.001) and surgeon level (R = 0.84, p < 0.001). Similarly, crowdworker ratings of the renal artery dissection were highly correlated with expert assessments (R = 0.83, p < 0.001) for the unique surgery-specific assessment question.
  
  
  CONCLUSIONS
  We conclude that crowdsourced assessment of qualitative performance ratings may be an alternative and/or adjunct to surgical experts' ratings and would provide a rapid scalable solution to triage technical skills.
authors.affiliation: 1 Department of Urology, Tulane University School of Medicine , New Orleans, Louisiana.
authors.forename: Mary K
authors.initials: MK
authors.lastname: Powers
authors.affiliation: 1 Department of Urology, Tulane University School of Medicine , New Orleans, Louisiana.
authors.forename: Aaron
authors.initials: A
authors.lastname: Boonjindasup
authors.affiliation: 1 Department of Urology, Tulane University School of Medicine , New Orleans, Louisiana.
authors.forename: Michael
authors.initials: M
authors.lastname: Pinsky
authors.affiliation: 1 Department of Urology, Tulane University School of Medicine , New Orleans, Louisiana.
authors.forename: Philip
authors.initials: P
authors.lastname: Dorsey
authors.affiliation: 1 Department of Urology, Tulane University School of Medicine , New Orleans, Louisiana.
authors.forename: Michael
authors.initials: M
authors.lastname: Maddox
authors.affiliation: 2 Department of Urology, University of Florida School of Medicine , Gainesville, Florida.
authors.forename: Li-Ming
authors.initials: LM
authors.lastname: Su
authors.affiliation: 3 Mayo Clinic , Rochester, Minnesota.
authors.forename: Matthew
authors.initials: M
authors.lastname: Gettman
authors.affiliation: 4 Department of Urology, Indiana University School of Medicine , Indianapolis, Indiana.
authors.forename: Chandru P
authors.initials: CP
authors.lastname: Sundaram
authors.affiliation: 5 Department of Urology, Mayo Clinic , Scottsdale, Arizona.
authors.forename: Erik P
authors.initials: EP
authors.lastname: Castle
authors.affiliation: 6 Department of Urology, St. Michael's Hospital University of Toronto , Toronto, Canada .
authors.forename: Jason Y
authors.initials: JY
authors.lastname: Lee
authors.affiliation: 1 Department of Urology, Tulane University School of Medicine , New Orleans, Louisiana.
authors.forename: Benjamin R
authors.initials: BR
authors.lastname: Lee
mesh_terms: D002983:Clinical Competence; D063045:Crowdsourcing; D006801:Humans; D009392:Nephrectomy / Q000193:education*; D012077:Renal Artery; D012082:Renal Veins; D015203:Reproducibility of Results; D065287:Robotic Surgical Procedures; D014741:Video Recording