title: Comparative analysis of data collection methods for individualized modeling of radiologists' visual similarity judgments in mammograms.
abstract: OBJECTIVE
  We conducted an observer study to investigate how the data collection method affects the efficacy of modeling individual radiologists' judgments regarding the perceptual similarity of breast masses on mammograms.
  
  
  METHODS
  Six observers of varying experience levels in breast imaging were recruited to assess the perceptual similarity of mammographic masses. The observers' subjective judgments were collected using (i) a rating method, (ii) a preference method, and (iii) a hybrid method combining rating and ranking. Personalized user models were developed with the collected data to predict observers' opinions. The relative efficacy of each data collection method was assessed based on the classification accuracy of the resulting user models.
  
  
  RESULTS
  The average accuracy of the user models derived from data collected with the hybrid method was 55.5 ± 1.5%. The models were significantly more accurate (P < .0005) than those derived from the rating (45.3 ± 3.5%) and the preference (40.8 ± 5%) methods. On average, the rating data collection method was significantly faster than the other two methods (P < .0001). No time advantage was observed between the preference and the hybrid methods.
  
  
  CONCLUSIONS
  A hybrid method combining rating and ranking is an intuitive and efficient way for collecting subjective similarity judgments to model human perceptual opinions with a higher accuracy than other, more commonly used data collection methods.
authors.affiliation: Biomedical Science and Engineering Center, Oak Ridge National Laboratory, 1 Bethel Valley Road, P.O. Box 2008, Oak Ridge, TN 37831. Electronic address: tourassig@ornl.gov.
authors.forename: Georgia
authors.initials: G
authors.lastname: Tourassi
authors.forename: Hong-Jun
authors.initials: HJ
authors.lastname: Yoon
authors.forename: Songhua
authors.initials: S
authors.lastname: Xu
authors.forename: Garnetta
authors.initials: G
authors.lastname: Morin-Ducote
authors.forename: Kathy
authors.initials: K
authors.lastname: Hudson
mesh_terms: D001943:Breast Neoplasms / Q000000981:diagnostic imaging*; D003625:Data Collection / Q000379:methods*; D005260:Female; D006801:Humans; D007600:Judgment; D008327:Mammography / Q000706:statistics & numerical data*; D015588:Observer Variation
keywords: Breast imaging; mammography; observer variability; perception; visual similarity user modeling