title: Crowdsourcing for error detection in cortical surface delineations.
abstract: OBJECTIVE
  With the recent trend toward big data analysis, neuroimaging datasets have grown substantially in the past years. While larger datasets potentially offer important insights for medical research, one major bottleneck is the requirement for resources of medical experts needed to validate automatic processing results. To address this issue, the goal of this paper was to assess whether anonymous nonexperts from an online community can perform quality control of MR-based cortical surface delineations derived by an automatic algorithm.
  
  
  METHODS
  So-called knowledge workers from an online crowdsourcing platform were asked to annotate errors in automatic cortical surface delineations on 100 central, coronal slices of MR images.
  
  
  RESULTS
  On average, annotations for 100 images were obtained in less than an hour. When using expert annotations as reference, the crowd on average achieves a sensitivity of 82 % and a precision of 42 %. Merging multiple annotations per image significantly improves the sensitivity of the crowd (up to 95 %), but leads to a decrease in precision (as low as 22 %).
  
  
  CONCLUSIONS
  Our experiments show that the detection of errors in automatic cortical surface delineations generated by anonymous untrained workers is feasible. Future work will focus on increasing the sensitivity of our method further, such that the error detection tasks can be handled exclusively by the crowd and expert resources can be focused on error correction.
authors.affiliation: Neurobiology Research Unit, Center for Integrated Molecular Brain Imaging, Rigshospitalet, Juliane Maries Vej 28, 2100, Copenhagen, Denmark. melanie.ganz@nru.dk.
authors.forename: Melanie
authors.identifier: http://orcid.org/0000-0002-9120-8098
authors.initials: M
authors.lastname: Ganz
authors.affiliation: Pallas Ludens GmbH, Im Bosseldorn 29, 69126, Heidelberg, Germany.
authors.forename: Daniel
authors.initials: D
authors.lastname: Kondermann
authors.affiliation: Pallas Ludens GmbH, Im Bosseldorn 29, 69126, Heidelberg, Germany.
authors.forename: Jonas
authors.initials: J
authors.lastname: Andrulis
authors.affiliation: Neurobiology Research Unit, Center for Integrated Molecular Brain Imaging, Rigshospitalet, Juliane Maries Vej 28, 2100, Copenhagen, Denmark.
authors.forename: Gitte Moos
authors.initials: GM
authors.lastname: Knudsen
authors.affiliation: Computer-assisted Interventions, German Cancer Research Center (DKFZ), Im Neuenheimer Feld 280, 69120, Heidelberg, Germany.
authors.forename: Lena
authors.initials: L
authors.lastname: Maier-Hein
mesh_terms: D000465:Algorithms; D001331:Automation / Q000592:standards; D002540:Cerebral Cortex / Q000000981:diagnostic imaging*; D063045:Crowdsourcing / Q000379:methods*; D066264:Datasets as Topic; D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000592:standards*; D021621:Imaging, Three-Dimensional; D008279:Magnetic Resonance Imaging; D059906:Neuroimaging; D011786:Quality Control
keywords: Cortical surface; Crowdsourcing; FreeSurfer; Neuroimaging