title: Building a gold standard to construct search filters: a case study with biomarkers for oral cancer.
abstract: OBJECTIVE
  To support clinical researchers, librarians and informationists may need search filters for particular tasks. Development of filters typically depends on a "gold standard" dataset. This paper describes generalizable methods for creating a gold standard to support future filter development and evaluation using oral squamous cell carcinoma (OSCC) as a case study. OSCC is the most common malignancy affecting the oral cavity. Investigation of biomarkers with potential prognostic utility is an active area of research in OSCC. The methods discussed here should be useful for designing quality search filters in similar domains.
  
  
  METHODS
  The authors searched MEDLINE for prognostic studies of OSCC, developed annotation guidelines for screeners, ran three calibration trials before annotating the remaining body of citations, and measured inter-annotator agreement (IAA).
  
  
  RESULTS
  We retrieved 1,818 citations. After calibration, we screened the remaining citations (n = 1,767; 97.2%); IAA was substantial (kappa = 0.76). The dataset has 497 (27.3%) citations representing OSCC studies of potential prognostic biomarkers.
  
  
  CONCLUSIONS
  The gold standard dataset is likely to be high quality and useful for future development and evaluation of filters for OSCC studies of potential prognostic biomarkers.
  
  
  CONCLUSIONS
  The methodology we used is generalizable to other domains requiring a reference standard to evaluate the performance of search filters. A gold standard is essential because the labels regarding relevance enable computation of diagnostic metrics, such as sensitivity and specificity. Librarians and informationists with data analysis skills could contribute to developing gold standard datasets and subsequent filters tuned for their patrons' domains of interest.
authors.affiliation: jjf60@pitt.edu , Fellow of the American Academy of Oral and Maxillofacial Pathology, Diplomate of the American Board of Oral and Maxillofacial Pathology, and National Library of Medicine Fellow; cds51@pitt.edu , Researcher; tseytlin@pitt.edu , Systems Developer; (Featured), tcb24@pitt.edu , Assistant Professor, Department of Biomedical Informatics and Department of Dental Public Health; School of Medicine and School of Dental Medicine, University of Pittsburgh, 5607 Baum Boulevard, Suite 514, Pittsburgh, PA 15206-3701.
authors.forename: John J
authors.initials: JJ
authors.lastname: Frazier
authors.affiliation: jjf60@pitt.edu , Fellow of the American Academy of Oral and Maxillofacial Pathology, Diplomate of the American Board of Oral and Maxillofacial Pathology, and National Library of Medicine Fellow; cds51@pitt.edu , Researcher; tseytlin@pitt.edu , Systems Developer; (Featured), tcb24@pitt.edu , Assistant Professor, Department of Biomedical Informatics and Department of Dental Public Health; School of Medicine and School of Dental Medicine, University of Pittsburgh, 5607 Baum Boulevard, Suite 514, Pittsburgh, PA 15206-3701.
authors.forename: Corey D
authors.initials: CD
authors.lastname: Stein
authors.affiliation: jjf60@pitt.edu , Fellow of the American Academy of Oral and Maxillofacial Pathology, Diplomate of the American Board of Oral and Maxillofacial Pathology, and National Library of Medicine Fellow; cds51@pitt.edu , Researcher; tseytlin@pitt.edu , Systems Developer; (Featured), tcb24@pitt.edu , Assistant Professor, Department of Biomedical Informatics and Department of Dental Public Health; School of Medicine and School of Dental Medicine, University of Pittsburgh, 5607 Baum Boulevard, Suite 514, Pittsburgh, PA 15206-3701.
authors.forename: Eugene
authors.initials: E
authors.lastname: Tseytlin
authors.affiliation: jjf60@pitt.edu , Fellow of the American Academy of Oral and Maxillofacial Pathology, Diplomate of the American Board of Oral and Maxillofacial Pathology, and National Library of Medicine Fellow; cds51@pitt.edu , Researcher; tseytlin@pitt.edu , Systems Developer; (Featured), tcb24@pitt.edu , Assistant Professor, Department of Biomedical Informatics and Department of Dental Public Health; School of Medicine and School of Dental Medicine, University of Pittsburgh, 5607 Baum Boulevard, Suite 514, Pittsburgh, PA 15206-3701.
authors.forename: Tanja
authors.initials: T
authors.lastname: Bekhuis
mesh_terms: D014408:Biomarkers, Tumor / Q000145:classification*; D016206:Databases, Bibliographic; D006801:Humans; D033181:Information Dissemination; D016247:Information Storage and Retrieval / Q000379:methods / Q000592:standards*; D016239:MEDLINE; D046650:Medical Subject Headings; D009062:Mouth Neoplasms / Q000175:diagnosis*; D019982:Organizational Case Studies; D010506:Periodicals as Topic / Q000706:statistics & numerical data*; D012015:Reference Standards
chemical_list: D014408:Biomarkers, Tumor