title: Plus Disease in Retinopathy of Prematurity: Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis.
abstract: To determine expert agreement on relative retinopathy of prematurity (ROP) disease severity and whether computer-based image analysis can model relative disease severity, and to propose consideration of a more continuous severity score for ROP.
  
  
  
  We developed 2 databases of clinical images of varying disease severity (100 images and 34 images) as part of the Imaging and Informatics in ROP (i-ROP) cohort study and recruited expert physician, nonexpert physician, and nonphysician graders to classify and perform pairwise comparisons on both databases.
  
  
  
  Six participating expert ROP clinician-scientists, each with a minimum of 10 years of clinical ROP experience and 5 ROP publications, and 5 image graders (3 physicians and 2 nonphysician graders) who analyzed images that were obtained during routine ROP screening in neonatal intensive care units.
  
  
  
  Images in both databases were ranked by average disease classification (classification ranking), by pairwise comparison using the Elo rating method (comparison ranking), and by correlation with the i-ROP computer-based image analysis system.
  
  
  
  Interexpert agreement (weighted κ statistic) compared with the correlation coefficient (CC) between experts on pairwise comparisons and correlation between expert rankings and computer-based image analysis modeling.
  
  
  
  There was variable interexpert agreement on diagnostic classification of disease (plus, preplus, or normal) among the 6 experts (mean weighted κ, 0.27; range, 0.06-0.63), but good correlation between experts on comparison ranking of disease severity (mean CC, 0.84; range, 0.74-0.93) on the set of 34 images. Comparison ranking provided a severity ranking that was in good agreement with ranking obtained by classification ranking (CC, 0.92). Comparison ranking on the larger dataset by both expert and nonexpert graders demonstrated good correlation (mean CC, 0.97; range, 0.95-0.98). The i-ROP system was able to model this continuous severity with good correlation (CC, 0.86).
  
  
  
  Experts diagnose plus disease on a continuum, with poor absolute agreement on classification but good relative agreement on disease severity. These results suggest that the use of pairwise rankings and a continuous severity score, such as that provided by the i-ROP system, may improve agreement on disease severity in the future.
authors: Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, Massachusetts.
authors: Jayashree
authors: J
authors: Kalpathy-Cramer
authors: Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland, Oregon.
authors: J Peter
authors: JP
authors: Campbell
authors: Cognitive Systems Laboratory, Northeastern University, Boston, Massachusetts.
authors: Deniz
authors: D
authors: Erdogmus
authors: Cognitive Systems Laboratory, Northeastern University, Boston, Massachusetts.
authors: Peng
authors: P
authors: Tian
authors: Cognitive Systems Laboratory, Northeastern University, Boston, Massachusetts.
authors: Dharanish
authors: D
authors: Kedarisetti
authors: Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland, Oregon.
authors: Chace
authors: C
authors: Moleta
authors: Department of Ophthalmology, Ross Eye Institute, State University of New York at Buffalo, Buffalo, New York.
authors: James D
authors: JD
authors: Reynolds
authors: Department of Ophthalmology, Sidra Medical & Research Center, Doha, Qatar.
authors: Kelly
authors: K
authors: Hutcheson
authors: Retina Consultants, Chicago, Illinois.
authors: Michael J
authors: MJ
authors: Shapiro
authors: Wilmer Eye Institute, Johns Hopkins University School of Medicine, Baltimore, Maryland.
authors: Michael X
authors: MX
authors: Repka
authors: Long Island Vitreoretinal Consultants, Great Neck, New York.
authors: Philip
authors: P
authors: Ferrone
authors: Associated Retinal Consultants, Oakland University, Royal Oak, Michigan.
authors: Kimberly
authors: K
authors: Drenser
authors: Department of Ophthalmology, Columbia University, New York, New York.
authors: Jason
authors: J
authors: Horowitz
authors: Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon.
authors: Kemal
authors: K
authors: Sonmez
authors: Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon.
authors: Ryan
authors: R
authors: Swan
authors: Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland, Oregon.
authors: Susan
authors: S
authors: Ostmo
authors: Department of Ophthalmology and Visual Sciences, Illinois Eye and Ear Infirmary, University of Illinois at Chicago, Chicago, Illinois.
authors: Karyn E
authors: KE
authors: Jonas
authors: Department of Ophthalmology and Visual Sciences, Illinois Eye and Ear Infirmary, University of Illinois at Chicago, Chicago, Illinois.
authors: R V Paul
authors: RV
authors: Chan
authors: Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland, Oregon; Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon. Electronic address: chiangm@ohsu.edu.
authors: Michael F
authors: MF
authors: Chiang
mesh_terms: D002983:Clinical Competence; D003941:Diagnostic Techniques, Ophthalmological / Q000639:trends*; D006801:Humans; D007091:Image Processing, Computer-Assisted / Q000379:methods*; D007231:Infant, Newborn; D012372:ROC Curve; D015203:Reproducibility of Results; D012160:Retina / Q000000981:diagnostic imaging*; D012178:Retinopathy of Prematurity / Q000145:classification / Q000175:diagnosis*; D012720:Severity of Illness Index