title: Development and Evaluation of Reference Standards for Image-based Telemedicine Diagnosis and Clinical Research Studies in Ophthalmology.
abstract: Information systems managing image-based data for telemedicine or clinical research applications require a reference standard representing the correct diagnosis. Accurate reference standards are difficult to establish because of imperfect agreement among physicians, and discrepancies between clinical vs. image-based diagnosis. This study is designed to describe the development and evaluation of reference standards for image-based diagnosis, which combine diagnostic impressions of multiple image readers with the actual clinical diagnoses. We show that agreement between image reading and clinical examinations was imperfect (689 [32%] discrepancies in 2148 image readings), as was inter-reader agreement (kappa 0.490-0.652). This was improved by establishing an image-based reference standard defined as the majority diagnosis given by three readers (13% discrepancies with image readers). It was further improved by establishing an overall reference standard that incorporated the clinical diagnosis (10% discrepancies with image readers). These principles of establishing reference standards may be applied to improve robustness of real-world systems supporting image-based diagnosis.
authors.affiliation: Departments of Ophthalmology & Clinical Epidemiology, Oregon Health & Science University, Portland, OR.
authors.forename: Michael C
authors.initials: MC
authors.lastname: Ryan
authors.affiliation: Departments of Ophthalmology & Clinical Epidemiology, Oregon Health & Science University, Portland, OR.
authors.forename: Susan
authors.initials: S
authors.lastname: Ostmo
authors.affiliation: Department of Ophthalmology, Weill Cornell Medical College, New York, NY.
authors.forename: Karyn
authors.initials: K
authors.lastname: Jonas
authors.affiliation: Department of Ophthalmology, University of Miami, Miami, FL.
authors.forename: Audina
authors.initials: A
authors.lastname: Berrocal
authors.affiliation: Department of Ophthalmology, William Beaumont Hospital, Royal Oak, MI.
authors.forename: Kimberly
authors.initials: K
authors.lastname: Drenser
authors.affiliation: Department of Ophthalmology, Columbia University, New York,NY.
authors.forename: Jason
authors.initials: J
authors.lastname: Horowitz
authors.affiliation: Department of Ophthalmology, Children's Hospital Los Angeles, Los Angeles, CA.
authors.forename: Thomas C
authors.initials: TC
authors.lastname: Lee
authors.affiliation: Department of Pediatrics, Cedars-Sinai Medical Center, Los Angeles, CA.
authors.forename: Charles
authors.initials: C
authors.lastname: Simmons
authors.affiliation: Department of Ophthalmology, APEC, Mexico City, Mexico.
authors.forename: Maria-Ana
authors.initials: MA
authors.lastname: Martinez-Castellanos
authors.affiliation: Department of Ophthalmology, Weill Cornell Medical College, New York, NY.
authors.forename: R V Paul
authors.initials: RV
authors.lastname: Chan
authors.affiliation: Departments of Ophthalmology & Clinical Epidemiology, Oregon Health & Science University, Portland, OR ; Departments of Medical Informatics & Clinical Epidemiology, Oregon Health & Science University, Portland, OR.
authors.forename: Michael F
authors.initials: MF
authors.lastname: Chiang
mesh_terms: D035843:Biomedical Research; D003952:Diagnostic Imaging / Q000592:standards*; D005654:Fundus Oculi; D006801:Humans; D007231:Infant, Newborn; D020407:Internet; D015588:Observer Variation; D009885:Ophthalmology / Q000592:standards*; D012015:Reference Standards; D012178:Retinopathy of Prematurity / Q000175:diagnosis*; D017216:Telemedicine / Q000592:standards*