title: Automated early detection of diabetic retinopathy.
abstract: OBJECTIVE
  To compare the performance of automated diabetic retinopathy (DR) detection, using the algorithm that won the 2009 Retinopathy Online Challenge Competition in 2009, the Challenge2009, against that of the one currently used in EyeCheck, a large computer-aided early DR detection project.
  
  
  METHODS
  Evaluation of diagnostic test or technology.
  
  
  METHODS
  Fundus photographic sets, consisting of 2 fundus images from each eye, were evaluated from 16670 patient visits of 16,670 people with diabetes who had not previously been diagnosed with DR.
  
  
  METHODS
  The fundus photographic set from each visit was analyzed by a single retinal expert; 793 of the 16,670 sets were classified as containing more than minimal DR (threshold for referral). The outcomes of the 2 algorithmic detectors were applied separately to the dataset and were compared by standard statistical measures.
  
  
  METHODS
  The area under the receiver operating characteristic curve (AUC), a measure of the sensitivity and specificity of DR detection.
  
  
  RESULTS
  Agreement was high, and examination results indicating more than minimal DR were detected with an AUC of 0.839 by the EyeCheck algorithm and an AUC of 0.821 for the Challenge2009 algorithm, a statistically nonsignificant difference (z-score, 1.91). If either of the algorithms detected DR in combination, the AUC for detection was 0.86, the same as the theoretically expected maximum. At 90% sensitivity, the specificity of the EyeCheck algorithm was 47.7% and that of the Challenge2009 algorithm was 43.6%.
  
  
  CONCLUSIONS
  Diabetic retinopathy detection algorithms seem to be maturing, and further improvements in detection performance cannot be differentiated from best clinical practices, because the performance of competitive algorithm development now has reached the human intrareader variability limit. Additional validation studies on larger, well-defined, but more diverse populations of patients with diabetes are needed urgently, anticipating cost-effective early detection of DR in millions of people with diabetes to triage those patients who need further care at a time when they have early rather than advanced DR.
authors.affiliation: Department of Ophthalmology and Visual Sciences, University of Iowa Hospitals and Clinics, Iowa City, Iowa 52242, USA.
authors.forename: Michael D
authors.initials: MD
authors.lastname: Abràmoff
authors.forename: Joseph M
authors.initials: JM
authors.lastname: Reinhardt
authors.forename: Stephen R
authors.initials: SR
authors.lastname: Russell
authors.forename: James C
authors.initials: JC
authors.lastname: Folk
authors.forename: Vinit B
authors.initials: VB
authors.lastname: Mahajan
authors.forename: Meindert
authors.initials: M
authors.lastname: Niemeijer
authors.forename: Gwénolé
authors.initials: G
authors.lastname: Quellec
mesh_terms: D000368:Aged; D000465:Algorithms; D019540:Area Under Curve; D003930:Diabetic Retinopathy / Q000175:diagnosis*; D003936:Diagnosis, Computer-Assisted; D003941:Diagnostic Techniques, Ophthalmological; D005260:Female; D006801:Humans; D008297:Male; D015588:Observer Variation; D012372:ROC Curve; D012680:Sensitivity and Specificity
references.citation: Comput Biomed Res. 1996 Aug;29(4):284-302
references.pmid: 8812075
references.citation: Ophthalmology. 2006 May;113(5):826-32
references.pmid: 16650679
references.citation: IEEE Trans Med Imaging. 2004 Apr;23(4):501-9
references.pmid: 15084075
references.citation: IEEE Trans Med Imaging. 2010 Jan;29(1):185-95
references.pmid: 19822469
references.citation: Conf Proc IEEE Eng Med Biol Soc. 2006;2006:4432-5
references.pmid: 17947087
references.citation: Prim Care. 1995 Jun;22(2):213-33
references.pmid: 7617782
references.citation: Invest Ophthalmol Vis Sci. 2003 Feb;44(2):767-71
references.pmid: 12556412
references.citation: Annu Int Conf IEEE Eng Med Biol Soc. 2008;2008:5433-6
references.pmid: 19163946
references.citation: IEEE Trans Med Imaging. 2009 May;28(5):775-85
references.pmid: 19150786
references.citation: IEEE Trans Med Imaging. 2007 Jan;26(1):116-27
references.pmid: 17243590
references.citation: Br J Ophthalmol. 2007 Nov;91(11):1518-23
references.pmid: 17585001
references.citation: Ophthalmology. 1991 May;98(5 Suppl):786-806
references.pmid: 2062513
references.citation: Med Image Anal. 2006 Dec;10(6):888-98
references.pmid: 17138215
references.citation: Invest Ophthalmol Vis Sci. 2003 Feb;44(2):761-6
references.pmid: 12556411
references.citation: IEEE Trans Med Imaging. 2006 Sep;25(9):1223-32
references.pmid: 16967807
references.citation: Diabet Med. 2003 Jul;20(7):528-34
references.pmid: 12823232
references.citation: JAMA. 2001 Sep 12;286(10):1195-200
references.pmid: 11559264
references.citation: Med Biol Eng Comput. 2002 Jan;40(1):2-13
references.pmid: 11954703
references.citation: IEEE Trans Med Imaging. 2005 May;24(5):584-92
references.pmid: 15889546
references.citation: Arch Ophthalmol. 2007 Mar;125(3):406-10
references.pmid: 17353416
references.citation: Annu Int Conf IEEE Eng Med Biol Soc. 2008;2008:3538-41
references.pmid: 19163472
references.citation: Ophthalmology. 2000 Jan;107(1):19-24
references.pmid: 10647713
references.citation: Br J Ophthalmol. 1977 Mar;61(3):192-201
references.pmid: 851521
references.citation: Am J Ophthalmol. 2002 Aug;134(2):204-13
references.pmid: 12140027
references.citation: Ophthalmology. 1991 May;98(5 Suppl):823-33
references.pmid: 2062515
references.citation: Curr Opin Ophthalmol. 2006 Dec;17(6):519-22
references.pmid: 17065919
references.citation: Eye (Lond). 1997;11 ( Pt 5):622-8
references.pmid: 9474307
references.citation: Annu Int Conf IEEE Eng Med Biol Soc. 2008;2008:5441-4
references.pmid: 19163948
references.citation: Invest Ophthalmol Vis Sci. 1992 May;33(6):1888-93
references.pmid: 1582794
references.citation: Health Aff (Millwood). 2002 Jan-Feb;21(1):140-54
references.pmid: 11900066
references.citation: Ophthalmology. 1991 May;98(5 Suppl):766-85
references.pmid: 2062512
references.citation: Invest Ophthalmol Vis Sci. 2007 May;48(5):2260-7
references.pmid: 17460289
references.citation: Telemed J E Health. 2005 Dec;11(6):668-74
references.pmid: 16430386
references.citation: IEEE Trans Med Imaging. 2008 Sep;27(9):1230-41
references.pmid: 18779064
references.citation: IEEE Trans Med Imaging. 2002 Oct;21(10):1236-43
references.pmid: 12585705
references.citation: Diabet Med. 2000 Aug;17(8):588-94
references.pmid: 11073180
references.citation: Diabetes Care. 2008 Feb;31(2):193-8
references.pmid: 18024852
references.citation: Diabetes Care. 2000 Mar;23(3):390-404
references.pmid: 10868871
references.citation: Br J Ophthalmol. 2007 Nov;91(11):1512-7
references.pmid: 17504851
references.citation: Biometrics. 1997 Mar;53(1):370-82
references.pmid: 9147602
references.citation: J Clin Pathol. 1996 Jul;49(7):597-9
references.pmid: 8813964
references.citation: Comput Biol Med. 1998 May;28(3):225-38
references.pmid: 9784961