title: Evaluation of a system for automatic detection of diabetic retinopathy from color fundus photographs in a large population of patients with diabetes.
abstract: OBJECTIVE
  To evaluate the performance of a system for automated detection of diabetic retinopathy in digital retinal photographs, built from published algorithms, in a large, representative, screening population.
  
  
  METHODS
  We conducted a retrospective analysis of 10,000 consecutive patient visits, specifically exams (four retinal photographs, two left and two right) from 5,692 unique patients from the EyeCheck diabetic retinopathy screening project imaged with three types of cameras at 10 centers. Inclusion criteria included no previous diagnosis of diabetic retinopathy, no previous visit to ophthalmologist for dilated eye exam, and both eyes photographed. One of three retinal specialists evaluated each exam as unacceptable quality, no referable retinopathy, or referable retinopathy. We then selected exams with sufficient image quality and determined presence or absence of referable retinopathy. Outcome measures included area under the receiver operating characteristic curve (number needed to miss one case [NNM]) and type of false negative.
  
  
  RESULTS
  Total area under the receiver operating characteristic curve was 0.84, and NNM was 80 at a sensitivity of 0.84 and a specificity of 0.64. At this point, 7,689 of 10,000 exams had sufficient image quality, 4,648 of 7,689 (60%) were true negatives, 59 of 7,689 (0.8%) were false negatives, 319 of 7,689 (4%) were true positives, and 2,581 of 7,689 (33%) were false positives. Twenty-seven percent of false negatives contained large hemorrhages and/or neovascularizations.
  
  
  CONCLUSIONS
  Automated detection of diabetic retinopathy using published algorithms cannot yet be recommended for clinical practice. However, performance is such that evaluation on validated, publicly available datasets should be pursued. If algorithms can be improved, such a system may in the future lead to improved prevention of blindness and vision loss in patients with diabetes.
authors.affiliation: Retina Service, Department of Ophthalmology and Visual Sciences, University of Iowa Hospitals and Clinics, Iowa City, Iowa, USA. michael-abramoff@uiowa.edu
authors.forename: Michael D
authors.initials: MD
authors.lastname: Abràmoff
authors.forename: Meindert
authors.initials: M
authors.lastname: Niemeijer
authors.forename: Maria S A
authors.initials: MS
authors.lastname: Suttorp-Schulten
authors.forename: Max A
authors.initials: MA
authors.lastname: Viergever
authors.forename: Stephen R
authors.initials: SR
authors.lastname: Russell
authors.forename: Bram
authors.initials: B
authors.lastname: van Ginneken
mesh_terms: D000465:Algorithms; D003920:Diabetes Mellitus / Q000503:physiopathology*; D003930:Diabetic Retinopathy / Q000175:diagnosis*; D005128:Eye Diseases / Q000175:diagnosis; D005451:Fluorescein Angiography; D006801:Humans; D018579:Patient Selection; D015203:Reproducibility of Results; D012160:Retina / Q000503:physiopathology; D012189:Retrospective Studies; D012680:Sensitivity and Specificity