title: An observational study to assess if automated diabetic retinopathy image assessment software can replace one or more steps of manual imaging grading and to determine their cost-effectiveness.
abstract: Diabetic retinopathy screening in England involves labour-intensive manual grading of retinal images. Automated retinal image analysis systems (ARIASs) may offer an alternative to manual grading.
  
  
  
  To determine the screening performance and cost-effectiveness of ARIASs to replace level 1 human graders or pre-screen with ARIASs in the NHS diabetic eye screening programme (DESP). To examine technical issues associated with implementation.
  
  
  
  Observational retrospective measurement comparison study with a real-time evaluation of technical issues and a decision-analytic model to evaluate cost-effectiveness.
  
  
  
  A NHS DESP.
  
  
  
  Consecutive diabetic patients who attended a routine annual NHS DESP visit.
  
  
  
  Retinal images were manually graded and processed by three ARIASs: iGradingM (version 1.1; originally Medalytix Group Ltd, Manchester, UK, but purchased by Digital Healthcare, Cambridge, UK, at the initiation of the study, purchased in turn by EMIS Health, Leeds, UK, after conclusion of the study), Retmarker (version 0.8.2, Retmarker Ltd, Coimbra, Portugal) and EyeArt (Eyenuk Inc., Woodland Hills, CA, USA). The final manual grade was used as the reference standard. Arbitration on a subset of discrepancies between manual grading and the use of an ARIAS by a reading centre masked to all grading was used to create a reference standard manual grade modified by arbitration.
  
  
  
  Screening performance (sensitivity, specificity, false-positive rate and likelihood ratios) and diagnostic accuracy [95% confidence intervals (CIs)] of ARIASs. A secondary analysis explored the influence of camera type and patients' ethnicity, age and sex on screening performance. Economic analysis estimated the cost per appropriate screening outcome identified.
  
  
  
  A total of 20,258 patients with 102,856 images were entered into the study. The sensitivity point estimates of the ARIASs were as follows: EyeArt 94.7% (95% CI 94.2% to 95.2%) for any retinopathy, 93.8% (95% CI 92.9% to 94.6%) for referable retinopathy and 99.6% (95% CI 97.0% to 99.9%) for proliferative retinopathy; and Retmarker 73.0% (95% CI 72.0% to 74.0%) for any retinopathy, 85.0% (95% CI 83.6% to 86.2%) for referable retinopathy and 97.9% (95% CI 94.9 to 99.1%) for proliferative retinopathy. iGradingM classified all images as either 'disease' or 'ungradable', limiting further iGradingM analysis. The sensitivity and false-positive rates for EyeArt were not affected by ethnicity, sex or camera type but sensitivity declined marginally with increasing patient age. The screening performance of Retmarker appeared to vary with patient's age, ethnicity and camera type. Both EyeArt and Retmarker were cost saving relative to manual grading either as a replacement for level 1 human grading or used prior to level 1 human grading, although the latter was less cost-effective. A threshold analysis testing the highest ARIAS cost per patient before which ARIASs became more expensive per appropriate outcome than human grading, when used to replace level 1 grader, was Retmarker £3.82 and EyeArt £2.71 per patient.
  
  
  
  The non-randomised study design limited the health economic analysis but the same retinal images were processed by all ARIASs in this measurement comparison study.
  
  
  
  Retmarker and EyeArt achieved acceptable sensitivity for referable retinopathy and false-positive rates (compared with human graders as reference standard) and appear to be cost-effective alternatives to a purely manual grading approach. Future work is required to develop technical specifications to optimise deployment and address potential governance issues.
  
  
  
  The National Institute for Health Research (NIHR) Health Technology Assessment programme, a Fight for Sight Grant (Hirsch grant award) and the Department of Health's NIHR Biomedical Research Centre for Ophthalmology at Moorfields Eye Hospital and the University College London Institute of Ophthalmology.
authors.affiliation: National Institute for Health Research Moorfields Biomedical Research Centre, Moorfields Eye Hospital, London, UK.
authors.forename: Adnan
authors.initials: A
authors.lastname: Tufail
authors.affiliation: Population Health Research Institute, St George's, University of London, London, UK.
authors.forename: Venediktos V
authors.initials: VV
authors.lastname: Kapetanakis
authors.affiliation: Department of Social Policy, LSE Health, London School of Economics and Political Science, London, UK.
authors.forename: Sebastian
authors.initials: S
authors.lastname: Salas-Vega
authors.affiliation: National Institute for Health Research Moorfields Biomedical Research Centre, Moorfields Eye Hospital, London, UK.
authors.forename: Catherine
authors.initials: C
authors.lastname: Egan
authors.affiliation: Department of Social Policy, LSE Health, London School of Economics and Political Science, London, UK.
authors.forename: Caroline
authors.initials: C
authors.lastname: Rudisill
authors.affiliation: Population Health Research Institute, St George's, University of London, London, UK.
authors.forename: Christopher G
authors.initials: CG
authors.lastname: Owen
authors.affiliation: National Institute for Health Research Moorfields Biomedical Research Centre, Moorfields Eye Hospital, London, UK.
authors.forename: Aaron
authors.initials: A
authors.lastname: Lee
authors.affiliation: National Institute for Health Research Moorfields Biomedical Research Centre, Moorfields Eye Hospital, London, UK.
authors.forename: Vern
authors.initials: V
authors.lastname: Louw
authors.affiliation: Homerton University Hospital Foundation Trust, London, UK.
authors.forename: John
authors.initials: J
authors.lastname: Anderson
authors.affiliation: National Institute for Health Research Moorfields Biomedical Research Centre, Moorfields Eye Hospital, London, UK.
authors.forename: Gerald
authors.initials: G
authors.lastname: Liew
authors.affiliation: Homerton University Hospital Foundation Trust, London, UK.
authors.forename: Louis
authors.initials: L
authors.lastname: Bolter
authors.affiliation: Bristol Eye Hospital, Bristol, UK.
authors.forename: Clare
authors.initials: C
authors.lastname: Bailey
authors.affiliation: Doheny Eye Institute, Los Angeles, CA, USA.
authors.forename: SriniVas
authors.initials: S
authors.lastname: Sadda
authors.affiliation: Centre for Health Informatics & Multiprofessional Education (CHIME), Institute of Health Informatics, University College London, London, UK.
authors.forename: Paul
authors.initials: P
authors.lastname: Taylor
authors.affiliation: Population Health Research Institute, St George's, University of London, London, UK.
authors.forename: Alicja R
authors.initials: AR
authors.lastname: Rudnicka
mesh_terms: D000293:Adolescent; D000328:Adult; D000367:Age Factors; D000368:Aged; D000369:Aged, 80 and over; D002648:Child; D003362:Cost-Benefit Analysis; D003930:Diabetic Retinopathy / Q000175:diagnosis* / Q000208:ethnology / Q000473:pathology; D004739:England; D005006:Ethnicity; D005189:False Positive Reactions; D005260:Female; D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000191:economics* / Q000295:instrumentation / Q000379:methods*; D008297:Male; D008403:Mass Screening / Q000379:methods* / Q000592:standards; D008875:Middle Aged; D012189:Retrospective Studies; D012680:Sensitivity and Specificity; D012984:Software; D013222:State Medicine; D013673:Technology Assessment, Biomedical; D055815:Young Adult