title: Reliability and type of consumer health documents on the World Wide Web: an annotation study.
abstract: BACKGROUND
  In this paper we present a detailed scheme for annotating medical web pages designed for health care consumers. The annotation is along two axes: first, by reliability (the extent to which the medical information on the page can be trusted), second, by the type of page (patient leaflet, commercial, link, medical article, testimonial, or support).
  
  
  RESULTS
  We analyze inter-rater agreement among three judges for each axis. Inter-rater agreement was moderate (0.77 accuracy, 0.62 F-measure, 0.49 Kappa) on the page reliability axis and good (0.81 accuracy, 0.72 F-measure, 0.73 Kappa) along the page type axis.
  
  
  CONCLUSIONS
  We have shown promising results in this study that appropriate classes of pages can be developed and used by human annotators to annotate web pages with reasonable to good agreement.
  
  
  BACKGROUND
  No.
authors: Computer Science Department, California State University, Stanislaus, One University Circle, Turlock California, 95382, USA. mmartin@cs.csustan.edu.
authors: Melanie J
authors: MJ
authors: Martin