Screening for amblyopia in preverbal children with photoscreening photographs: IV. Interobserver variability in photograph grading: origin and method of reduction.
BACKGROUND
Earlier detection of childhood vision disorders is a prominent goal of vision screening. The Medical Technology and Innovation (MTI) PhotoScreener addresses this objective. Use of this camera does not require verbal feedback and may be administered early in a child's development. Decreasing the variability in photograph grading results will boost the utility of any photoscreening system. This report aims to understand and to decrease intra- and inter-observer variability in grading photoscreening photographs.


METHODS
We dissected the photograph grading process and quantified the intra- and inter-observer agreement using intraclass plot correlation coefficients. We evaluated the outcome of a two grader verification system vs. adjudicated measurements with Receiver Operator Characteristic (ROC) curves.


METHODS
Data on 955 children under 5 years of age, normal except for refractive error, each with complete photoscreening and eye examination data, culled from two previous studies.


METHODS
Intra- and inter-observer agreement in measuring bright crescent dimensions and pupillary diameters. Sensitivity and specificity of detection of hyperopia.


RESULTS
Measurements of bright crescents are associated with greater variability than are measurements of pupillary diameters. Recognition and omission of light "rim" measurements from photograph grading will result in superior inter-observer agreement. Photograph independent errors increase variability and may be corrected by remeasurement. A verification system in which the most discrepant 5% of measurements are redone results in ROC curves similar to adjudicated dimension.


CONCLUSIONS
We conclude: 1) two novices grading photographs can do as well as one expert in most cases; 2) the proposed grading methodology has undergone statistical validation and can be used in other areas of ophthalmology and medicine; and 3) inter-observer variability, one of the limitations of photoscreening photograph grading, can be reduced. For 95% of the photographs, two novices achieve similar true positive and true negative values with or without adjudication.
Kim
Alan H
AH
Wilmer Opthamological Institute, Johns Hopkins University of Baltimore, MD, USA.
Chen
Jessica
J
Ottar-Pfeifer
Wanda
W
Lengauer
Barbara
B
Holgado
Sandra
S
Stager
David R
DR
Parks
Marshall M
MM
Beauchamp
George R
GR
Scott
William
W
Marsh
Marta J
MJ
Tong
Patrick Y
PY
D000550:Amblyopia / Q000175:diagnosis*; D002675:Child, Preschool; D005189:False Positive Reactions; D006801:Humans; D007223:Infant; D015588:Observer Variation; D010781:Photography / Q000379:methods / Q000592:standards*; D011237:Predictive Value of Tests; D012372:ROC Curve; D015203:Reproducibility of Results; D015351:Vision Screening / Q000379:methods / Q000592:standards*