title: Exploration of analysis methods for diagnostic imaging tests: problems with ROC AUC and confidence scores in CT colonography.
abstract: BACKGROUND
  Different methods of evaluating diagnostic performance when comparing diagnostic tests may lead to different results. We compared two such approaches, sensitivity and specificity with area under the Receiver Operating Characteristic Curve (ROC AUC) for the evaluation of CT colonography for the detection of polyps, either with or without computer assisted detection.
  
  
  METHODS
  In a multireader multicase study of 10 readers and 107 cases we compared sensitivity and specificity, using radiological reporting of the presence or absence of polyps, to ROC AUC calculated from confidence scores concerning the presence of polyps. Both methods were assessed against a reference standard. Here we focus on five readers, selected to illustrate issues in design and analysis. We compared diagnostic measures within readers, showing that differences in results are due to statistical methods.
  
  
  RESULTS
  Reader performance varied widely depending on whether sensitivity and specificity or ROC AUC was used. There were problems using confidence scores; in assigning scores to all cases; in use of zero scores when no polyps were identified; the bimodal non-normal distribution of scores; fitting ROC curves due to extrapolation beyond the study data; and the undue influence of a few false positive results. Variation due to use of different ROC methods exceeded differences between test results for ROC AUC.
  
  
  CONCLUSIONS
  The confidence scores recorded in our study violated many assumptions of ROC AUC methods, rendering these methods inappropriate. The problems we identified will apply to other detection studies using confidence scores. We found sensitivity and specificity were a more reliable and clinically appropriate method to compare diagnostic tests.
authors.affiliation: Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom.
authors.forename: Susan
authors.initials: S
authors.lastname: Mallett
authors.affiliation: Centre for Medical Imaging, University College London, London, United Kingdom.
authors.forename: Steve
authors.initials: S
authors.lastname: Halligan
authors.affiliation: Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom.
authors.forename: Gary S
authors.initials: GS
authors.lastname: Collins
authors.affiliation: Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom.
authors.forename: Doug G
authors.initials: DG
authors.lastname: Altman
mesh_terms: D019540:Area Under Curve; D003111:Colonic Polyps / Q000000981:diagnostic imaging*; D023881:Colonography, Computed Tomographic / Q000379:methods*; D003952:Diagnostic Imaging / Q000379:methods*; D006801:Humans; D012372:ROC Curve; D011857:Radiographic Image Interpretation, Computer-Assisted; D012680:Sensitivity and Specificity