title: ROC Curve for Extremely Subtle Lung Nodules on Chest Radiographs Confirmed by CT Scan.
abstract: OBJECTIVE
  The aim of the present study was to test the hypothesis that when a radiologist does not perceive an abnormality in images that contain either extremely subtle abnormalities or no abnormalities, the radiologist cannot distinguish these two types of images and the receiver operating characteristic (ROC) curve reflects that performance.
  
  
  METHODS
  This retrospective study was conducted with approval of our institutional review board. Four general radiologists participated in an observer performance study of 100 chest images, each of which had a 5 × 5 cm region of interest (ROI) drawn (50 containing a lung nodule, and 50 did not, based on computed tomography [CT] confirmation). About half of the lung nodules were extremely subtle. The readers reported their confidence that a nodule was present within the ROI, from which empirical and maximum-likelihood "proper" binormal and conventional binormal ROC curves were estimated. The readers also reported whether they saw an abnormality that could be a nodule within the ROI.
  
  
  RESULTS
  Empirical ROC curves deviated from typical ROC-curve shapes, and a portion of the curve leading to the northeast corner of the ROC space had relatively steep and constant slopes. The readers reported not seeing anything suggestive of a lung nodule in this portion of the ROC curve, which also corresponded to cases that either contained extremely subtle nodules or normal cases. The average area under the ROC curves (mean ± standard deviation) was 0.66 ± 0.02 for proper binormal, 0.62 ± 0.02 for conventional binormal, and 0.60 ± 0.03 for trapezoidal ROC curves.
  
  
  CONCLUSIONS
  When a radiologist does not perceive an abnormality in images that contain either extremely subtle abnormalities or no abnormalities, the ROC curve (or a portion thereof) is characterized by a straight line, which is not consistent with conventional ROC theories.
authors.affiliation: Department of Radiology, The University of Chicago, 5841 South Maryland Avenue, Chicago, IL 60637.
authors.forename: Hua
authors.initials: H
authors.lastname: Shang
authors.affiliation: Department of Radiology, The University of Chicago, 5841 South Maryland Avenue, Chicago, IL 60637. Electronic address: yjiang@uchicago.edu.
authors.forename: Yulei
authors.initials: Y
authors.lastname: Jiang
authors.affiliation: Department of Radiology, The University of Chicago, 5841 South Maryland Avenue, Chicago, IL 60637.
authors.forename: Feng
authors.initials: F
authors.lastname: Li
authors.affiliation: Department of Radiology, The University of Chicago, 5841 South Maryland Avenue, Chicago, IL 60637.
authors.forename: Heber
authors.initials: H
authors.lastname: MacMahon
authors.affiliation: Department of Radiology, The University of Chicago, 5841 South Maryland Avenue, Chicago, IL 60637.
authors.forename: Jianing
authors.initials: J
authors.lastname: Wang
mesh_terms: D006801:Humans; D016013:Likelihood Functions; D015588:Observer Variation; D011336:Probability; D012372:ROC Curve; D013902:Radiography, Thoracic / Q000706:statistics & numerical data*; D011871:Radiology / Q000592:standards; D012189:Retrospective Studies; D003074:Solitary Pulmonary Nodule / Q000000981:diagnostic imaging*; D014057:Tomography, X-Ray Computed / Q000706:statistics & numerical data*
keywords: Receiver operating characteristic (ROC) analysis; extremely subtle abnormality; human observer; ideal observer; observer performance