The Accuracy and Reliability of Crowdsource Annotations of Digital Retinal Images.
OBJECTIVE
Crowdsourcing is based on outsourcing computationally intensive tasks to numerous individuals in the online community who have no formal training. Our aim was to develop a novel online tool designed to facilitate large-scale annotation of digital retinal images, and to assess the accuracy of crowdsource grading using this tool, comparing it to expert classification.


METHODS
We used 100 retinal fundus photograph images with predetermined disease criteria selected by two experts from a large cohort study. The Amazon Mechanical Turk Web platform was used to drive traffic to our site so anonymous workers could perform a classification and annotation task of the fundus photographs in our dataset after a short training exercise. Three groups were assessed: masters only, nonmasters only and nonmasters with compulsory training. We calculated the sensitivity, specificity, and area under the curve (AUC) of receiver operating characteristic (ROC) plots for all classifications compared to expert grading, and used the Dice coefficient and consensus threshold to assess annotation accuracy.


RESULTS
In total, we received 5389 annotations for 84 images (excluding 16 training images) in 2 weeks. A specificity and sensitivity of 71% (95% confidence interval [CI], 69%-74%) and 87% (95% CI, 86%-88%) was achieved for all classifications. The AUC in this study for all classifications combined was 0.93 (95% CI, 0.91-0.96). For image annotation, a maximal Dice coefficient (∼0.6) was achieved with a consensus threshold of 0.25.


CONCLUSIONS
This study supports the hypothesis that annotation of abnormalities in retinal images by ophthalmologically naive individuals is comparable to expert annotation. The highest AUC and agreement with expert annotation was achieved in the nonmasters with compulsory training group.


CONCLUSIONS
The use of crowdsourcing as a technique for retinal image analysis may be comparable to expert graders and has the potential to deliver timely, accurate, and cost-effective image analysis.
NIHR Biomedical Research Centre at Moorfields Eye Hospital & UCL Institute of Ophthalmology, London, UK.
Danny
D
Mitry
VAMPIRE project, School of Science and Engineering, University of Dundee, Dundee, UK.
Kris
K
Zutis
Centre for Clinical Brain Sciences, University of Edinburgh and Princess Alexandra Eye Pavilion, Edinburgh, UK.
Baljean
B
Dhillon
NIHR Biomedical Research Centre at Moorfields Eye Hospital & UCL Institute of Ophthalmology, London, UK.
Tunde
T
Peto
Department of Public Health and Primary Care, University of Cambridge Strangeways Research Laboratory, Worts Causeway, Cambridge, UK.
Shabina
S
Hayat
Department of Clinical Gerontology, Addenbrookes Hospital, University of Cambridge, Cambridge, UK.
Kay-Tee
KT
Khaw
School of Optometry and Vision Sciences, Cardiff University, Cardiff, UK.
James E
JE
Morgan
Duncan of Jordanstone College of Arts and Design, University of Dundee, Dundee, UK.
Wendy
W
Moncur
VAMPIRE project, School of Science and Engineering, University of Dundee, Dundee, UK.
Emanuele
E
Trucco
NIHR Biomedical Research Centre at Moorfields Eye Hospital & UCL Institute of Ophthalmology, London, UK.
Paul J
PJ
Foster
crowdsourcing; image analysis; retina
PLoS One. 2013 Aug 21;8(8):e71154
23990935
J Am Coll Radiol. 2016 May;13(5):554-61
26924162
Med Image Comput Comput Assist Interv. 2014;17(Pt 2):438-45
25485409
J Surg Res. 2014 Mar;187(1):65-71
24555877
Invest Ophthalmol Vis Sci. 2011 Jul 01;52(7):4866-71
21527381
Invest Ophthalmol Vis Sci. 1981 Jul;21(1 Pt 2):1-226
7195893
J Rehabil Res Dev. 2006 Sep-Oct;43(6):733-40
17310422
PLoS One. 2013;8(3):e57410
23516406
Diabetes Care. 2004 May;27(5):1047-53
15111519
Perspect Psychol Sci. 2011 Jan;6(1):3-5
26162106
Curr Diab Rep. 2015 Mar;15(3):14
25697773
Br J Ophthalmol. 2015 Jan;99(1):64-8
25091950
Acad Radiol. 2004 Feb;11(2):178-89
14974593
PLoS One. 2015 Feb 18;10(2):e0117401
25692287
Ophthalmology. 2004 May;111(5):1055-62
15121388
JAMA Ophthalmol. 2013 Mar;131(3):351-7
23494039
PLoS One. 2012;7(5):e37245
22606353
J Med Internet Res. 2014 Oct 30;16(10):e233
25356929
BMJ Open. 2013 Mar 19;3(3):null
23516272
Radiology. 2012 Mar;262(3):824-33
22274839
Ophthalmology. 2010 Jun;117(6):1147-54
20399502