title: Screening for amblyopia in preverbal children with photoscreening photographs: IV. Interobserver variability in photograph grading: origin and method of reduction.
abstract: BACKGROUND
  Earlier detection of childhood vision disorders is a prominent goal of vision screening. The Medical Technology and Innovation (MTI) PhotoScreener addresses this objective. Use of this camera does not require verbal feedback and may be administered early in a child's development. Decreasing the variability in photograph grading results will boost the utility of any photoscreening system. This report aims to understand and to decrease intra- and inter-observer variability in grading photoscreening photographs.
  
  
  METHODS
  We dissected the photograph grading process and quantified the intra- and inter-observer agreement using intraclass plot correlation coefficients. We evaluated the outcome of a two grader verification system vs. adjudicated measurements with Receiver Operator Characteristic (ROC) curves.
  
  
  METHODS
  Data on 955 children under 5 years of age, normal except for refractive error, each with complete photoscreening and eye examination data, culled from two previous studies.
  
  
  METHODS
  Intra- and inter-observer agreement in measuring bright crescent dimensions and pupillary diameters. Sensitivity and specificity of detection of hyperopia.
  
  
  RESULTS
  Measurements of bright crescents are associated with greater variability than are measurements of pupillary diameters. Recognition and omission of light "rim" measurements from photograph grading will result in superior inter-observer agreement. Photograph independent errors increase variability and may be corrected by remeasurement. A verification system in which the most discrepant 5% of measurements are redone results in ROC curves similar to adjudicated dimension.
  
  
  CONCLUSIONS
  We conclude: 1) two novices grading photographs can do as well as one expert in most cases; 2) the proposed grading methodology has undergone statistical validation and can be used in other areas of ophthalmology and medicine; and 3) inter-observer variability, one of the limitations of photoscreening photograph grading, can be reduced. For 95% of the photographs, two novices achieve similar true positive and true negative values with or without adjudication.
authors.affiliation: Wilmer Opthamological Institute, Johns Hopkins University of Baltimore, MD, USA.
authors.forename: Alan H
authors.initials: AH
authors.lastname: Kim
authors.forename: Jessica
authors.initials: J
authors.lastname: Chen
authors.forename: Wanda
authors.initials: W
authors.lastname: Ottar-Pfeifer
authors.forename: Barbara
authors.initials: B
authors.lastname: Lengauer
authors.forename: Sandra
authors.initials: S
authors.lastname: Holgado
authors.forename: David R
authors.initials: DR
authors.lastname: Stager
authors.forename: Marshall M
authors.initials: MM
authors.lastname: Parks
authors.forename: George R
authors.initials: GR
authors.lastname: Beauchamp
authors.forename: William
authors.initials: W
authors.lastname: Scott
authors.forename: Marta J
authors.initials: MJ
authors.lastname: Marsh
authors.forename: Patrick Y
authors.initials: PY
authors.lastname: Tong
mesh_terms: D000550:Amblyopia / Q000175:diagnosis*; D002675:Child, Preschool; D005189:False Positive Reactions; D006801:Humans; D007223:Infant; D015588:Observer Variation; D010781:Photography / Q000379:methods / Q000592:standards*; D011237:Predictive Value of Tests; D012372:ROC Curve; D015203:Reproducibility of Results; D015351:Vision Screening / Q000379:methods / Q000592:standards*