title: Leveraging the crowd for annotation of retinal images.
abstract: Medical data presents a number of challenges. It tends to be unstructured, noisy and protected. To train algorithms to understand medical images, doctors can label the condition associated with a particular image, but obtaining enough labels can be difficult. We propose an annotation approach which starts with a small pool of expertly annotated images and uses their expertise to rate the performance of crowd-sourced annotations. In this paper we demonstrate how to apply our approach for annotation of large-scale datasets of retinal images. We introduce a novel data validation procedure which is designed to cope with noisy ground-truth data and with non-consistent input from both experts and crowd-workers.
authors.forename: George
authors.initials: G
authors.lastname: Leifman
authors.forename: Tristan
authors.initials: T
authors.lastname: Swedish
authors.forename: Karin
authors.initials: K
authors.lastname: Roesch
authors.forename: Ramesh
authors.initials: R
authors.lastname: Raskar
mesh_terms: D000465:Algorithms; D066289:Data Curation / Q000379:methods*; D016208:Databases, Factual; D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000379:methods*; D012160:Retina / Q000033:anatomy & histology*