title: Usability of quality measures for online health information: Can commonly used technical quality criteria be reliably assessed?
abstract: OBJECTIVE
  Many criteria have been developed to rate the quality of online health information. To effectively evaluate quality, consumers must use quality criteria that can be reliably assessed. However, few instruments have been validated for inter-rater agreement. Therefore, we assessed the degree to which two raters could reliably assess 22 popularly cited quality criteria on a sample of 42 complementary and alternative medicine Web sites.
  
  
  METHODS
  We determined the degree of inter-rater agreement by calculating the percentage agreement, Cohen's kappa, and prevalence- and bias-adjusted kappa (PABAK).
  
  
  RESULTS
  Our un-calibrated analysis showed poor inter-rater agreement on eight of the 22 quality criteria. Therefore, we created operational definitions for each of the criteria, decreased the number of assessment choices and defined where to look for the information. As a result 18 of the 22 quality criteria were reliably assessed (inter-rater agreement > or = 0.6).
  
  
  CONCLUSIONS
  We conclude that even with precise definitions, some commonly used quality criteria cannot be reliably assessed. However, inter-rater agreement can be improved with precise operational definitions.
authors.affiliation: School of Health Information Sciences, The University of Texas Health Science Center at Houston, 7000 Fannin Street, Suite 600, Houston, TX 77030, USA. elmer.v.bernstam@uth.tmc.edu
authors.forename: Elmer V
authors.initials: EV
authors.lastname: Bernstam
authors.forename: Smitha
authors.initials: S
authors.lastname: Sagaram
authors.forename: Muhammad
authors.initials: M
authors.lastname: Walji
authors.forename: Craig W
authors.initials: CW
authors.lastname: Johnson
authors.forename: Funda
authors.initials: F
authors.lastname: Meric-Bernstam
mesh_terms: D000529:Complementary Therapies; D006801:Humans; D020407:Internet; D008490:Medical Informatics; D011786:Quality Control