title: Establishing a gold standard for test sets: variation in interpretive agreement of expert mammographers.
abstract: OBJECTIVE
  Test sets for assessing and improving radiologic image interpretation have been used for decades and typically evaluate performance relative to gold standard interpretations by experts. To assess test sets for screening mammography, a gold standard for whether a woman should be recalled for additional workup is needed, given that interval cancers may be occult on mammography and some findings ultimately determined to be benign require additional imaging to determine if biopsy is warranted. Using experts to set a gold standard assumes little variation occurs in their interpretations, but this has not been explicitly studied in mammography.
  
  
  METHODS
  Using digitized films from 314 screening mammography exams (n = 143 cancer cases) performed in the Breast Cancer Surveillance Consortium, we evaluated interpretive agreement among three expert radiologists who independently assessed whether each examination should be recalled, and the lesion location, finding type (mass, calcification, asymmetric density, or architectural distortion), and interpretive difficulty in the recalled images.
  
  
  RESULTS
  Agreement among the three expert pairs for recall/no recall was higher for cancer cases (mean 74.3 ± 6.5) than for noncancers (mean 62.6 ± 7.1). Complete agreement on recall, lesion location, finding type and difficulty ranged from 36.4% to 42.0% for cancer cases and from 43.9% to 65.6% for noncancer cases. Two of three experts agreed on recall and lesion location for 95.1% of cancer cases and 91.8% of noncancer cases, but all three experts agreed on only 55.2% of cancer cases and 42.1% of noncancer cases.
  
  
  CONCLUSIONS
  Variability in expert interpretive is notable. A minimum of three independent experts combined with a consensus should be used for establishing any gold standard interpretation for test sets, especially for noncancer cases.
authors.affiliation: Department of Community & Family Medicine, Norris Cotton Cancer Center, Lebanon, NH, USA. Tracy.L.Onega@dartmouth.edu
authors.forename: Tracy
authors.initials: T
authors.lastname: Onega
authors.forename: Melissa L
authors.initials: ML
authors.lastname: Anderson
authors.forename: Diana L
authors.initials: DL
authors.lastname: Miglioretti
authors.forename: Diana S M
authors.initials: DS
authors.lastname: Buist
authors.forename: Berta
authors.initials: B
authors.lastname: Geller
authors.forename: Andy
authors.initials: A
authors.lastname: Bogart
authors.forename: Robert A
authors.initials: RA
authors.lastname: Smith
authors.forename: Edward A
authors.initials: EA
authors.lastname: Sickles
authors.forename: Barbara
authors.initials: B
authors.lastname: Monsees
authors.forename: Lawrence
authors.initials: L
authors.lastname: Bassett
authors.forename: Patricia A
authors.initials: PA
authors.lastname: Carney
authors.forename: Karla
authors.initials: K
authors.lastname: Kerlikowske
authors.forename: Bonnie C
authors.initials: BC
authors.lastname: Yankaskas
mesh_terms: D000328:Adult; D000368:Aged; D001943:Breast Neoplasms / Q000000981:diagnostic imaging* / Q000453:epidemiology*; D005260:Female; D006801:Humans; D008327:Mammography / Q000592:standards* / Q000706:statistics & numerical data*; D008875:Middle Aged; D015588:Observer Variation; D015995:Prevalence; D011361:Professional Competence / Q000592:standards*; D012015:Reference Standards; D012042:Registries; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D014481:United States / Q000453:epidemiology