title: Crowdsourcing image annotation for nucleus detection and segmentation in computational pathology: evaluating experts, automated methods, and the crowd.
abstract: The development of tools in computational pathology to assist physicians and biomedical scientists in the diagnosis of disease requires access to high-quality annotated images for algorithm learning and evaluation. Generating high-quality expert-derived annotations is time-consuming and expensive. We explore the use of crowdsourcing for rapidly obtaining annotations for two core tasks in com- putational pathology: nucleus detection and nucleus segmentation. We designed and implemented crowdsourcing experiments using the CrowdFlower platform, which provides access to a large set of labor channel partners that accesses and manages millions of contributors worldwide. We obtained annotations from four types of annotators and compared concordance across these groups. We obtained: crowdsourced annotations for nucleus detection and segmentation on a total of 810 images; annotations using automated methods on 810 images; annotations from research fellows for detection and segmentation on 477 and 455 images, respectively; and expert pathologist-derived annotations for detection and segmentation on 80 and 63 images, respectively. For the crowdsourced annotations, we evaluated performance across a range of contributor skill levels (1, 2, or 3). The crowdsourced annotations (4,860 images in total) were completed in only a fraction of the time and cost required for obtaining annotations using traditional methods. For the nucleus detection task, the research fellow-derived annotations showed the strongest concordance with the expert pathologist- derived annotations (F-M =93.68%), followed by the crowd-sourced contributor levels 1,2, and 3 and the automated method, which showed relatively similar performance (F-M = 87.84%, 88.49%, 87.26%, and 86.99%, respectively). For the nucleus segmentation task, the crowdsourced contributor level 3-derived annotations, research fellow-derived annotations, and automated method showed the strongest concordance with the expert pathologist-derived annotations (F-M = 66.41%, 65.93%, and 65.36%, respectively), followed by the contributor levels 2 and 1 (60.89% and 60.87%, respectively). When the research fellows were used as a gold-standard for the segmentation task, all three con- tributor levels of the crowdsourced annotations significantly outperformed the automated method (F-M = 62.21%, 62.47%, and 65.15% vs. 51.92%). Aggregating multiple annotations from the crowd to obtain a consensus annotation resulted in the strongest performance for the crowd-sourced segmentation. For both detection and segmentation, crowd-sourced performance is strongest with small images (400 × 400 pixels) and degrades significantly with the use of larger images (600 × 600 and 800 × 800 pixels). We conclude that crowdsourcing to non-experts can be used for large-scale labeling microtasks in computational pathology and offers a new approach for the rapid generation of labeled images for algorithm development and evaluation.
authors.affiliation: Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, USA. hirshad@bidmc.harvard.edu.
authors.forename: H
authors.initials: H
authors.lastname: Irshad
authors.forename: L
authors.initials: L
authors.lastname: Montaser-Kouhsari
authors.forename: G
authors.initials: G
authors.lastname: Waltz
authors.forename: O
authors.initials: O
authors.lastname: Bucur
authors.forename: J A
authors.initials: JA
authors.lastname: Nowak
authors.forename: F
authors.initials: F
authors.lastname: Dong
authors.forename: N W
authors.initials: NW
authors.lastname: Knoblauch
authors.forename: A H
authors.initials: AH
authors.lastname: Beck
mesh_terms: D000465:Algorithms; D002292:Carcinoma, Renal Cell / Q000175:diagnosis / Q000473:pathology; D002467:Cell Nucleus / Q000473:pathology*; D019295:Computational Biology / Q000379:methods; D063045:Crowdsourcing / Q000379:methods*; D066289:Data Curation; D016208:Databases, Factual; D005104:Expert Testimony; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods; D007680:Kidney Neoplasms / Q000175:diagnosis / Q000473:pathology; D009369:Neoplasms / Q000175:diagnosis* / Q000473:pathology*; D010338:Pathology, Clinical / Q000706:statistics & numerical data*