title: Evaluation of a large-scale biomedical data annotation initiative.
abstract: BACKGROUND
  This study describes a large-scale manual re-annotation of data samples in the Gene Expression Omnibus (GEO), using variables and values derived from the National Cancer Institute thesaurus. A framework is described for creating an annotation scheme for various diseases that is flexible, comprehensive, and scalable. The annotation structure is evaluated by measuring coverage and agreement between annotators.
  
  
  RESULTS
  There were 12,500 samples annotated with approximately 30 variables, in each of six disease categories - breast cancer, colon cancer, inflammatory bowel disease (IBD), rheumatoid arthritis (RA), systemic lupus erythematosus (SLE), and Type 1 diabetes mellitus (DM). The annotators provided excellent variable coverage, with known values for over 98% of three critical variables: disease state, tissue, and sample type. There was 89% strict inter-annotator agreement and 92% agreement when using semantic and partial similarity measures.
  
  
  CONCLUSIONS
  We show that it is possible to perform manual re-annotation of a large repository in a reliable manner.
authors.affiliation: Departments of Biomedical & Health Informatics, University of Washington, Seattle, WA, USA. rlacson@dsg.harvard.edu
authors.forename: Ronilda
authors.initials: R
authors.lastname: Lacson
authors.forename: Erik
authors.initials: E
authors.lastname: Pitzer
authors.forename: Christian
authors.initials: C
authors.lastname: Hinske
authors.forename: Pedro
authors.initials: P
authors.lastname: Galante
authors.forename: Lucila
authors.initials: L
authors.lastname: Ohno-Machado
mesh_terms: D000465:Algorithms; D019295:Computational Biology / Q000379:methods*; D030541:Databases, Genetic; D020869:Gene Expression Profiling; D016247:Information Storage and Retrieval