title: Inference based on diagnostic measures from studies of new imaging devices.
abstract: OBJECTIVE
  Before using a new diagnostic imaging device regularly in a clinic, it should be studied using patients and radiologists. Often such studies report diagnostic performance in terms of sensitivity, specificity, area under the receiver operating characteristic curve (AUC), or differences thereof. In this report we look at how these studies differ from actual future clinical practice and how those differences may affect reported performance measures.
  
  
  METHODS
  We review signal detection (receiver operating characteristic) theory and decision theory. We compare diagnostic measures from several published studies in medical imaging and examine how they relate to theory and each other.
  
  
  RESULTS
  We see that clinical decisions can be modeled using signal detection and decision theories. Sensitivity and specificity are inextricably linked with clinical factors, such as prevalence and costs. Imaging devices are used in many different ways in clinical practice, so that sensitivities, specificities, and AUCs measured in studies of new diagnostic imaging devices will differ from those in actual future clinical use.
  
  
  CONCLUSIONS
  Measured sensitivities, specificities, and the directions of changes thereof are not necessarily consistent or reproducible across studies of new diagnostic devices. A change in the AUC, which should be independent of clinical costs or prevalence, is a consistent measure across similar studies, and a positive change in AUC is indicative of additional diagnostic information that will be available to radiologists in a future clinical environment.
authors: U.S. Food and Drug Administration, 10903 New Hampshire Ave, Building 62, Room 3102, Silver Spring, MD, 20993-0002, USA. frank.samuelson@fda.hhs.gov
authors: Frank W
authors: FW
authors: Samuelson
mesh_terms: D019540:Area Under Curve; D001943:Breast Neoplasms / Q000000981:diagnostic imaging*; D003937:Diagnosis, Differential; D003952:Diagnostic Imaging / Q000379:methods / Q000706:statistics & numerical data; D005260:Female; D006801:Humans; D008327:Mammography / Q000379:methods / Q000706:statistics & numerical data*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D012815:Signal Processing, Computer-Assisted