title: Crowdsourcing for reference correspondence generation in endoscopic images.
abstract: Computer-assisted minimally-invasive surgery (MIS) is often based on algorithms that require establishing correspondences between endoscopic images. However, reference annotations frequently required to train or validate a method are extremely difficult to obtain because they are typically made by a medical expert with very limited resources, and publicly available data sets are still far too small to capture the wide range of anatomical/scene variance. Crowdsourcing is a new trend that is based on outsourcing cognitive tasks to many anonymous untrained individuals from an online community. To our knowledge, this paper is the first to investigate the concept of crowdsourcing in the context of endoscopic video image annotation for computer-assisted MIS. According to our study on publicly available in vivo data with manual reference annotations, anonymous non-experts obtain a median annotation error of 2 px (n = 10,000). By applying cluster analysis to multiple annotations per correspondence, this error can be reduced to about 1 px, which is comparable to that obtained by medical experts (n = 500). We conclude that crowdsourcing is a viable method for generating high quality reference correspondences in endoscopic video images.
authors.forename: Lena
authors.initials: L
authors.lastname: Maier-Hein
authors.forename: Sven
authors.initials: S
authors.lastname: Mersmann
authors.forename: Daniel
authors.initials: D
authors.lastname: Kondermann
authors.forename: Christian
authors.initials: C
authors.lastname: Stock
authors.forename: Hannes Gotz
authors.initials: HG
authors.lastname: Kenngott
authors.forename: Alexandro
authors.initials: A
authors.lastname: Sanchez
authors.forename: Martin
authors.initials: M
authors.lastname: Wagner
authors.forename: Anas
authors.initials: A
authors.lastname: Preukschas
authors.forename: Anna-Laura
authors.initials: AL
authors.lastname: Wekerle
authors.forename: Stefanie
authors.initials: S
authors.lastname: Helfert
authors.forename: Sebastian
authors.initials: S
authors.lastname: Bodenstedt
authors.forename: Stefanie
authors.initials: S
authors.lastname: Speidel
mesh_terms: D000465:Algorithms; D053704:Capsule Endoscopy / Q000379:methods*; D063045:Crowdsourcing / Q000379:methods*; D006801:Humans; D007089:Image Enhancement / Q000379:methods*; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D015588:Observer Variation; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D025321:Surgery, Computer-Assisted / Q000379:methods*; D014584:User-Computer Interface