title: Crowdtruth validation: a new paradigm for validating algorithms that rely on image correspondences.
abstract: OBJECTIVE
  Feature tracking and 3D surface reconstruction are key enabling techniques to computer-assisted minimally invasive surgery. One of the major bottlenecks related to training and validation of new algorithms is the lack of large amounts of annotated images that fully capture the wide range of anatomical/scene variance in clinical practice. To address this issue, we propose a novel approach to obtaining large numbers of high-quality reference image annotations at low cost in an extremely short period of time.
  
  
  METHODS
  The concept is based on outsourcing the correspondence search to a crowd of anonymous users from an online community (crowdsourcing) and comprises four stages: (1) feature detection, (2) correspondence search via crowdsourcing, (3) merging multiple annotations per feature by fitting Gaussian finite mixture models, (4) outlier removal using the result of the clustering as input for a second annotation task.
  
  
  RESULTS
  On average, 10,000 annotations were obtained within 24 h at a cost of $100. The annotation of the crowd after clustering and before outlier removal was of expert quality with a median distance of about 1 pixel to a publically available reference annotation. The threshold for the outlier removal task directly determines the maximum annotation error, but also the number of points removed.
  
  
  CONCLUSIONS
  Our concept is a novel and effective method for fast, low-cost and highly accurate correspondence generation that could be adapted to various other applications related to large-scale data annotation in medical image computing and computer-assisted interventions.
authors.affiliation: Junior Group Computer-Assisted Interventions, German Cancer Research Center (DKFZ), Heidelberg, Germany, l.maier-hein@dkfz.de.
authors.forename: Lena
authors.initials: L
authors.lastname: Maier-Hein
authors.forename: Daniel
authors.initials: D
authors.lastname: Kondermann
authors.forename: Tobias
authors.initials: T
authors.lastname: Roß
authors.forename: Sven
authors.initials: S
authors.lastname: Mersmann
authors.forename: Eric
authors.initials: E
authors.lastname: Heim
authors.forename: Sebastian
authors.initials: S
authors.lastname: Bodenstedt
authors.forename: Hannes Götz
authors.initials: HG
authors.lastname: Kenngott
authors.forename: Alexandro
authors.initials: A
authors.lastname: Sanchez
authors.forename: Martin
authors.initials: M
authors.lastname: Wagner
authors.forename: Anas
authors.initials: A
authors.lastname: Preukschas
authors.forename: Anna-Laura
authors.initials: AL
authors.lastname: Wekerle
authors.forename: Stefanie
authors.initials: S
authors.lastname: Helfert
authors.forename: Keno
authors.initials: K
authors.lastname: März
authors.forename: Arianeb
authors.initials: A
authors.lastname: Mehrabi
authors.forename: Stefanie
authors.initials: S
authors.lastname: Speidel
authors.forename: Christian
authors.initials: C
authors.lastname: Stock
mesh_terms: D000465:Algorithms; D019985:Benchmarking; D006801:Humans; D019060:Minimally Invasive Surgical Procedures / Q000379:methods*; D025321:Surgery, Computer-Assisted / Q000379:methods*