title: Plus Disease in Retinopathy of Prematurity: A Continuous Spectrum of Vascular Abnormality as a Basis of Diagnostic Variability.
abstract: To identify patterns of interexpert discrepancy in plus disease diagnosis in retinopathy of prematurity (ROP).
  
  
  
  We developed 2 datasets of clinical images as part of the Imaging and Informatics in ROP study and determined a consensus reference standard diagnosis (RSD) for each image based on 3 independent image graders and the clinical examination results. We recruited 8 expert ROP clinicians to classify these images and compared the distribution of classifications between experts and the RSD.
  
  
  
  Eight participating experts with more than 10 years of clinical ROP experience and more than 5 peer-reviewed ROP publications who analyzed images obtained during routine ROP screening in neonatal intensive care units.
  
  
  
  Expert classification of images of plus disease in ROP.
  
  
  
  Interexpert agreement (weighted κ statistic) and agreement and bias on ordinal classification between experts (analysis of variance [ANOVA]) and the RSD (percent agreement).
  
  
  
  There was variable interexpert agreement on diagnostic classifications between the 8 experts and the RSD (weighted κ, 0-0.75; mean, 0.30). The RSD agreement ranged from 80% to 94% for the dataset of 100 images and from 29% to 79% for the dataset of 34 images. However, when images were ranked in order of disease severity (by average expert classification), the pattern of expert classification revealed a consistent systematic bias for each expert consistent with unique cut points for the diagnosis of plus disease and preplus disease. The 2-way ANOVA model suggested a highly significant effect of both image and user on the average score (dataset A: P < 0.05 and adjusted R2= 0.82; and dataset B: P < 0.05 and adjusted R2= 0.6615).
  
  
  
  There is wide variability in the classification of plus disease by ROP experts, which occurs because experts have different cut points for the amounts of vascular abnormality required for presence of plus and preplus disease. This has important implications for research, teaching, and patient care for ROP and suggests that a continuous ROP plus disease severity score may reflect more accurately the behavior of expert ROP clinicians and may better standardize classification in the future.
authors: Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland, Oregon.
authors: J Peter
authors: JP
authors: Campbell
authors: Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Charlestown, Massachusetts.
authors: Jayashree
authors: J
authors: Kalpathy-Cramer
authors: Cognitive Systems Laboratory, Northeastern University, Boston, Massachusetts.
authors: Deniz
authors: D
authors: Erdogmus
authors: Cognitive Systems Laboratory, Northeastern University, Boston, Massachusetts.
authors: Peng
authors: P
authors: Tian
authors: Cognitive Systems Laboratory, Northeastern University, Boston, Massachusetts.
authors: Dharanish
authors: D
authors: Kedarisetti
authors: Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland, Oregon.
authors: Chace
authors: C
authors: Moleta
authors: Department of Ophthalmology, Ross Eye Institute, State University of New York at Buffalo, Buffalo, New York.
authors: James D
authors: JD
authors: Reynolds
authors: Department of Ophthalmology, Sidra Medical & Research Center, Doha, Qatar.
authors: Kelly
authors: K
authors: Hutcheson
authors: Retina Consultants, Chicago, Illinois.
authors: Michael J
authors: MJ
authors: Shapiro
authors: Wilmer Institute, Johns Hopkins University School of Medicine, Baltimore, Maryland.
authors: Michael X
authors: MX
authors: Repka
authors: Long Island Vitreoretinal Consultants, Great Neck, New York.
authors: Philip
authors: P
authors: Ferrone
authors: Associated Retinal Consultants, Oakland University, Royal Oak, Michigan.
authors: Kimberly
authors: K
authors: Drenser
authors: Department of Ophthalmology, Columbia University, New York, New York.
authors: Jason
authors: J
authors: Horowitz
authors: Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon.
authors: Kemal
authors: K
authors: Sonmez
authors: Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon.
authors: Ryan
authors: R
authors: Swan
authors: Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland, Oregon.
authors: Susan
authors: S
authors: Ostmo
authors: Department of Ophthalmology and Visual Sciences, Illinois Eye and Ear Infirmary, University of Illinois at Chicago, Chicago, Illinois.
authors: Karyn E
authors: KE
authors: Jonas
authors: Department of Ophthalmology and Visual Sciences, Illinois Eye and Ear Infirmary, University of Illinois at Chicago, Chicago, Illinois.
authors: R V Paul
authors: RV
authors: Chan
authors: Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland, Oregon; Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, Oregon. Electronic address: chiangm@ohsu.edu.
authors: Michael F
authors: MF
authors: Chiang
mesh_terms: D003937:Diagnosis, Differential; D005260:Female; D006801:Humans; D007231:Infant, Newborn; D008297:Male; D015997:Neonatal Screening / Q000379:methods*; D010781:Photography; D012372:ROC Curve; D015203:Reproducibility of Results; D012160:Retina / Q000000981:diagnostic imaging*; D012171:Retinal Vessels / Q000000981:diagnostic imaging*; D012178:Retinopathy of Prematurity / Q000145:classification / Q000175:diagnosis*