Evaluation of a computer-based system for plus disease diagnosis in retinopathy of prematurity.
OBJECTIVE
To measure accuracy and reliability of the computer-based Retinal Image Multiscale Analysis (RISA) system compared with those of recognized retinopathy of prematurity (ROP) experts, for plus disease diagnosis.


METHODS
Evaluation of diagnostic test or technology.


METHODS
Eleven recognized ROP experts and the RISA image analysis system interpreted a set of 20 wide-angle retinal photographs for presence of plus disease.


METHODS
All experts used a secure Web site to review independently 20 images for presence of plus disease. Images were also analyzed by measuring individual computer-based system parameters (integrated curvature [IC], diameter, and tortuosity index) for arterioles and venules and by computing linear combinations and logical combinations of those parameters. Performance was compared with a reference standard, defined as the majority vote of experts.


METHODS
Diagnostic accuracy was measured by calculating sensitivity, specificity, and receiver operating characteristic area under the curve (AUC) for plus disease diagnosis by each expert, and by each computer-based system parameter, compared with the reference standard. Diagnostic agreement was measured by calculating the mean kappa value of each expert compared with all other experts and the mean kappa value of each computer-based system parameter compared with all experts.


RESULTS
Among the 11 experts, sensitivity ranged from 0.167 to 1.000, specificity ranged from 0.714 to 1.000, AUC ranged from 0.798 to 1.000, and mean kappa compared with all other experts ranged from 0.288 to 0.689. Among individual computer system parameters, arteriolar IC had the highest diagnostic accuracy, with sensitivity of 1.000; specificity, 0.846; and AUC, 0.962. Arteriolar IC had the highest diagnostic agreement with experts, with a mean kappa value of 0.578.


CONCLUSIONS
A computer-based image analysis system has the potential to perform comparably to recognized ROP experts for plus disease diagnosis.
Department of Ophthalmology, Columbia University College of Physicians and Surgeons, New York, New York 10032, USA.
Susan
S
Koreen
Rony
R
Gelman
M Elena
ME
Martinez-Perez
Lei
L
Jiang
Audina M
AM
Berrocal
Ditte J
DJ
Hess
John T
JT
Flynn
Michael F
MF
Chiang
D001160:Arterioles / Q000473:pathology; D003936:Diagnosis, Computer-Assisted; D006801:Humans; D007091:Image Processing, Computer-Assisted; D007231:Infant, Newborn; D007234:Infant, Premature; D015588:Observer Variation; D010781:Photography; D011237:Predictive Value of Tests; D012372:ROC Curve; D015203:Reproducibility of Results; D012161:Retinal Artery / Q000473:pathology*; D012169:Retinal Vein / Q000473:pathology*; D012178:Retinopathy of Prematurity / Q000175:diagnosis*; D012680:Sensitivity and Specificity; D014699:Venules / Q000473:pathology