title: Can masses of non-experts train highly accurate image classifiers? A crowdsourcing approach to instrument segmentation in laparoscopic images.
abstract: Machine learning algorithms are gaining increasing interest in the context of computer-assisted interventions. One of the bottlenecks so far, however, has been the availability of training data, typically generated by medical experts with very limited resources. Crowdsourcing is a new trend that is based on outsourcing cognitive tasks to many anonymous untrained individuals from an online community. In this work, we investigate the potential of crowdsourcing for segmenting medical instruments in endoscopic image data. Our study suggests that (1) segmentations computed from annotations of multiple anonymous non-experts are comparable to those made by medical experts and (2) training data generated by the crowd is of the same quality as that annotated by medical experts. Given the speed of annotation, scalability and low costs, this implies that the scientific community might no longer need to rely on experts to generate reference or training data for certain applications. To trigger further research in endoscopic image processing, the data used in this study will be made publicly available.
authors.forename: Lena
authors.initials: L
authors.lastname: Maier-Hein
authors.forename: Sven
authors.initials: S
authors.lastname: Mersmann
authors.forename: Daniel
authors.initials: D
authors.lastname: Kondermann
authors.forename: Sebastian
authors.initials: S
authors.lastname: Bodenstedt
authors.forename: Alexandro
authors.initials: A
authors.lastname: Sanchez
authors.forename: Christian
authors.initials: C
authors.lastname: Stock
authors.forename: Hannes Gotz
authors.initials: HG
authors.lastname: Kenngott
authors.forename: Mathias
authors.initials: M
authors.lastname: Eisenmann
authors.forename: Stefanie
authors.initials: S
authors.lastname: Speidel
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D063045:Crowdsourcing / Q000295:instrumentation* / Q000379:methods*; D004867:Equipment Design; D019544:Equipment Failure Analysis; D006801:Humans; D007089:Image Enhancement / Q000295:instrumentation / Q000379:methods; D016247:Information Storage and Retrieval / Q000379:methods*; D020706:Laparoscopes; D010535:Laparoscopy / Q000379:methods*; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity