title: Potential effect of different radiologist reporting methods on studies showing benefit of CAD.
abstract: OBJECTIVE
  To investigate the effect of different reporting methods and performance measures on the assessment of the benefit of computer-aided diagnosis (CAD) in characterizing malignant and benign breast lesions on mammography and sonography.
  
  
  METHODS
  In a previous study, 10 observers provided three types of reporting data (probability of malignancy [PM] estimates, Breast Imaging Reporting and Data System [BI-RADS] ratings, and biopsy decisions), both without and with CAD. The current study compares alternative performance measures computed from the three types of reporting data. The area under the receiver operating characteristic curve (AUC) was computed from both the PM estimates and the BI-RADS ratings, whereas sensitivity and specificity were computed from all three data types. Sensitivity and specificity values calculated from either the PM estimates or the BI-RADS ratings were determined by setting both constant and user-dependent thresholds. Student's t-tests were used to evaluate the statistical significance of the differences in the performance measures without and with CAD.
  
  
  RESULTS
  The average AUC values of the 10 observers calculated from either PM estimates or BI-RADS ratings demonstrated statistically significant improvements in performance with CAD, increasing from 0.87 to 0.92 or 0.93, respectively. However, the statistical significance of improvements in sensitivity or specificity depended on the type of reporting data used.
  
  
  CONCLUSIONS
  Use of different types of reporting data in the computation of sensitivity and specificity may result in different conclusions concerning the benefit of CAD. Meaningful determination of sensitivity and specificity from PM estimates require the use of user-dependent thresholds.
authors.affiliation: University of Chicago, Department of Radiology, 5841 South Maryland Avenue, Chicago, IL 60637, USA. kjhorsch@uchicago.edu
authors.forename: Karla
authors.initials: K
authors.lastname: Horsch
authors.forename: Maryellen L
authors.initials: ML
authors.lastname: Giger
authors.forename: Charles E
authors.initials: CE
authors.lastname: Metz
mesh_terms: D019540:Area Under Curve; D001706:Biopsy; D001943:Breast Neoplasms / Q000000981:diagnostic imaging*; D003936:Diagnosis, Computer-Assisted / Q000706:statistics & numerical data*; D004282:Documentation / Q000706:statistics & numerical data*; D006801:Humans; D008327:Mammography; D015588:Observer Variation; D012372:ROC Curve; D012680:Sensitivity and Specificity; D016217:Ultrasonography, Mammary