title: Screening mammography: test set data can reasonably describe actual clinical reporting.
abstract: OBJECTIVE
  To establish the extent to which test set reading can represent actual clinical reporting in screening mammography.
  
  
  METHODS
  Institutional ethics approval was granted, and informed consent was obtained from each participating screen reader. The need for informed consent with respect to the use of patient materials was waived. Two hundred mammographic examinations were selected from examinations reported by 10 individual expert screen readers, resulting in 10 reader-specific test sets. Data generated from actual clinical reports were compared with three test set conditions: clinical test set reading with prior images, laboratory test set reading with prior images, and laboratory test set reading without prior images. A further set of five expert screen readers was asked to interpret a common set of images in two identical test set conditions to establish a baseline for intraobserver variability. Confidence scores (from 1 to 4) were assigned to the respective decisions made by readers. Region-of-interest (ROI) figures of merit (FOMs) and side-specific sensitivity and specificity were described for the actual clinical reporting of each reader-specific test set and were compared with those for the three test set conditions. Agreement between pairs of readings was performed by using the Kendall coefficient of concordance.
  
  
  RESULTS
  Moderate or acceptable levels of agreement were evident (W = 0.69-0.73, P < .01) when describing group performance between actual clinical reporting and test set conditions that were reasonably close to the established baseline (W = 0.77, P < .01) and were lowest when prior images were excluded. Higher median values for ROI FOMs were demonstrated for the test set conditions than for the actual clinical reporting values; this was possibly linked to changes in sensitivity.
  
  
  CONCLUSIONS
  Reasonable levels of agreement between actual clinical reporting and test set conditions can be achieved, although inflated sensitivity may be evident with test set conditions.
authors.affiliation: Medical Image Optimisation and Perception Group, Discipline of Medical Radiation Sciences (C42 Cumberland Campus, University of Sydney, East Street, Room M221, Sydney, NSW 2141, Australia. bsoh6456@uni.sydney.edu.au
authors.forename: BaoLin P
authors.initials: BP
authors.lastname: Soh
authors.forename: Warwick
authors.initials: W
authors.lastname: Lee
authors.forename: Mark F
authors.initials: MF
authors.lastname: McEntee
authors.forename: Peter L
authors.initials: PL
authors.lastname: Kench
authors.forename: Warren M
authors.initials: WM
authors.lastname: Reed
authors.forename: Rob
authors.initials: R
authors.lastname: Heard
authors.forename: Dev P
authors.initials: DP
authors.lastname: Chakraborty
authors.forename: Patrick C
authors.initials: PC
authors.lastname: Brennan
mesh_terms: D001943:Breast Neoplasms / Q000000981:diagnostic imaging*; D003657:Decision Making; D003937:Diagnosis, Differential; D005260:Female; D006801:Humans; D008327:Mammography; D015588:Observer Variation; D011361:Professional Competence; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity