title: Inter-observer agreement for quality measures applied to online health information.
abstract: Many quality criteria have been developed to rate the quality of online health information. However, few instruments have been validated for inter-observer reliability. Therefore, we assessed the degree to which two raters agree upon the presence or absence of information based on 22 popularly cited quality criteria on a sample of 21 complementary and alternative medicine websites. Our preliminary analysis showed a poor inter-rater agreement on 10 out of the 22 quality criteria. Therefore, we created operational definitions for each of the criteria, decreased the allowed choices and defined a location to look for the information. As a result 15 out of the 22 quality criteria had a kappa >0.6. We conclude that even with precise definitions some commonly used quality criteria to assess the quality of health information online cannot be reliably assessed. However, inter-rater agreement can be improved by providing precise operational definitions.
authors.affiliation: School of Health Information Sciences, The University of Texas Health Science Center at Houston, 77030, USA.
authors.forename: Smitha
authors.initials: S
authors.lastname: Sagaram
authors.forename: Muhammad
authors.initials: M
authors.lastname: Walji
authors.forename: Funda
authors.initials: F
authors.lastname: Meric-Bernstam
authors.forename: Craig
authors.initials: C
authors.lastname: Johnson
authors.forename: Elmer
authors.initials: E
authors.lastname: Bernstam
mesh_terms: D000529:Complementary Therapies; D006266:Health Education / Q000592:standards*; D007255:Information Services / Q000592:standards*; D020407:Internet / Q000592:standards; D015588:Observer Variation; D019984:Quality Indicators, Health Care