Automated early detection of diabetic retinopathy.
OBJECTIVE
To compare the performance of automated diabetic retinopathy (DR) detection, using the algorithm that won the 2009 Retinopathy Online Challenge Competition in 2009, the Challenge2009, against that of the one currently used in EyeCheck, a large computer-aided early DR detection project.


METHODS
Evaluation of diagnostic test or technology.


METHODS
Fundus photographic sets, consisting of 2 fundus images from each eye, were evaluated from 16670 patient visits of 16,670 people with diabetes who had not previously been diagnosed with DR.


METHODS
The fundus photographic set from each visit was analyzed by a single retinal expert; 793 of the 16,670 sets were classified as containing more than minimal DR (threshold for referral). The outcomes of the 2 algorithmic detectors were applied separately to the dataset and were compared by standard statistical measures.


METHODS
The area under the receiver operating characteristic curve (AUC), a measure of the sensitivity and specificity of DR detection.


RESULTS
Agreement was high, and examination results indicating more than minimal DR were detected with an AUC of 0.839 by the EyeCheck algorithm and an AUC of 0.821 for the Challenge2009 algorithm, a statistically nonsignificant difference (z-score, 1.91). If either of the algorithms detected DR in combination, the AUC for detection was 0.86, the same as the theoretically expected maximum. At 90% sensitivity, the specificity of the EyeCheck algorithm was 47.7% and that of the Challenge2009 algorithm was 43.6%.


CONCLUSIONS
Diabetic retinopathy detection algorithms seem to be maturing, and further improvements in detection performance cannot be differentiated from best clinical practices, because the performance of competitive algorithm development now has reached the human intrareader variability limit. Additional validation studies on larger, well-defined, but more diverse populations of patients with diabetes are needed urgently, anticipating cost-effective early detection of DR in millions of people with diabetes to triage those patients who need further care at a time when they have early rather than advanced DR.
Abràmoff
Michael D
MD
Department of Ophthalmology and Visual Sciences, University of Iowa Hospitals and Clinics, Iowa City, Iowa 52242, USA.
Reinhardt
Joseph M
JM
Russell
Stephen R
SR
Folk
James C
JC
Mahajan
Vinit B
VB
Niemeijer
Meindert
M
Quellec
Gwénolé
G
D000368:Aged; D000465:Algorithms; D019540:Area Under Curve; D003930:Diabetic Retinopathy / Q000175:diagnosis*; D003936:Diagnosis, Computer-Assisted; D003941:Diagnostic Techniques, Ophthalmological; D005260:Female; D006801:Humans; D008297:Male; D015588:Observer Variation; D012372:ROC Curve; D012680:Sensitivity and Specificity
Comput Biomed Res. 1996 Aug;29(4):284-302
8812075
Ophthalmology. 2006 May;113(5):826-32
16650679
IEEE Trans Med Imaging. 2004 Apr;23(4):501-9
15084075
IEEE Trans Med Imaging. 2010 Jan;29(1):185-95
19822469
Conf Proc IEEE Eng Med Biol Soc. 2006;2006:4432-5
17947087
Prim Care. 1995 Jun;22(2):213-33
7617782
Invest Ophthalmol Vis Sci. 2003 Feb;44(2):767-71
12556412
Annu Int Conf IEEE Eng Med Biol Soc. 2008;2008:5433-6
19163946
IEEE Trans Med Imaging. 2009 May;28(5):775-85
19150786
IEEE Trans Med Imaging. 2007 Jan;26(1):116-27
17243590
Br J Ophthalmol. 2007 Nov;91(11):1518-23
17585001
Ophthalmology. 1991 May;98(5 Suppl):786-806
2062513
Med Image Anal. 2006 Dec;10(6):888-98
17138215
Invest Ophthalmol Vis Sci. 2003 Feb;44(2):761-6
12556411
IEEE Trans Med Imaging. 2006 Sep;25(9):1223-32
16967807
Diabet Med. 2003 Jul;20(7):528-34
12823232
JAMA. 2001 Sep 12;286(10):1195-200
11559264
Med Biol Eng Comput. 2002 Jan;40(1):2-13
11954703
IEEE Trans Med Imaging. 2005 May;24(5):584-92
15889546
Arch Ophthalmol. 2007 Mar;125(3):406-10
17353416
Annu Int Conf IEEE Eng Med Biol Soc. 2008;2008:3538-41
19163472
Ophthalmology. 2000 Jan;107(1):19-24
10647713
Br J Ophthalmol. 1977 Mar;61(3):192-201
851521
Am J Ophthalmol. 2002 Aug;134(2):204-13
12140027
Ophthalmology. 1991 May;98(5 Suppl):823-33
2062515
Curr Opin Ophthalmol. 2006 Dec;17(6):519-22
17065919
Eye (Lond). 1997;11 ( Pt 5):622-8
9474307
Annu Int Conf IEEE Eng Med Biol Soc. 2008;2008:5441-4
19163948
Invest Ophthalmol Vis Sci. 1992 May;33(6):1888-93
1582794
Health Aff (Millwood). 2002 Jan-Feb;21(1):140-54
11900066
Ophthalmology. 1991 May;98(5 Suppl):766-85
2062512
Invest Ophthalmol Vis Sci. 2007 May;48(5):2260-7
17460289
Telemed J E Health. 2005 Dec;11(6):668-74
16430386
IEEE Trans Med Imaging. 2008 Sep;27(9):1230-41
18779064
IEEE Trans Med Imaging. 2002 Oct;21(10):1236-43
12585705
Diabet Med. 2000 Aug;17(8):588-94
11073180
Diabetes Care. 2008 Feb;31(2):193-8
18024852
Diabetes Care. 2000 Mar;23(3):390-404
10868871
Br J Ophthalmol. 2007 Nov;91(11):1512-7
17504851
Biometrics. 1997 Mar;53(1):370-82
9147602
J Clin Pathol. 1996 Jul;49(7):597-9
8813964
Comput Biol Med. 1998 May;28(3):225-38
9784961