title: Quantifying the clinical relevance of a laboratory observer performance paradigm.
abstract: OBJECTIVE
  Laboratory observer performance measurements, receiver operating characteristic (ROC) and free-response ROC (FROC) differ from actual clinical interpretations in several respects, which could compromise their clinical relevance. The objective of this study was to develop a method for quantifying the clinical relevance of a laboratory paradigm and apply it to compare the ROC and FROC paradigms in a nodule detection task.
  
  
  METHODS
  The original prospective interpretations of 80 digital chest radiographs were classified by the truth panel as correct (C=1) or incorrect (C=0), depending on correlation with additional imaging, and the average of C was interpreted as the clinical figure of merit. FROC data were acquired for 21 radiologists and ROC data were inferred using the highest ratings. The areas under the ROC and alternative FROC curves were used as laboratory figures of merit. Bootstrap analysis was conducted to estimate conventional agreement measures between laboratory and clinical figures of merit. Also computed was a pseudovalue-based image-level correctness measure of the laboratory interpretations, whose association with C as measured by the area (rAUC) under an appropriately defined relevance ROC curve, is as a measure of the clinical relevance of a laboratory paradigm.
  
  
  RESULTS
  Low correlations (e.g. κ=0.244) and near chance level rAUC values (e.g. 0.598), attributable to differences between the clinical and laboratory paradigms, were observed. The absolute width of the confidence interval was 0.38 for the interparadigm differences of the conventional measures and 0.14 for the difference of the rAUCs.
  
  
  CONCLUSIONS
  The rAUC measure was consistent with the traditional measures but was more sensitive to the differences in clinical relevance. A new relevance ROC method for quantifying the clinical relevance of a laboratory paradigm is proposed.
authors.affiliation: Department of Radiology, University of Pittsburgh, Pittsburgh, PA 15213, USA. dpc10@pitt.edu
authors.forename: D P
authors.initials: DP
authors.lastname: Chakraborty
authors.forename: T M
authors.initials: TM
authors.lastname: Haygood
authors.forename: J
authors.initials: J
authors.lastname: Ryan
authors.forename: E M
authors.initials: EM
authors.lastname: Marom
authors.forename: M
authors.initials: M
authors.lastname: Evanoff
authors.forename: M F
authors.initials: MF
authors.lastname: McEntee
authors.forename: P C
authors.initials: PC
authors.lastname: Brennan
mesh_terms: D000328:Adult; D000368:Aged; D000465:Algorithms; D006801:Humans; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D008175:Lung Neoplasms / Q000000981:diagnostic imaging*; D008297:Male; D008875:Middle Aged; D015588:Observer Variation; D012372:ROC Curve; D011856:Radiographic Image Enhancement / Q000379:methods*; D013902:Radiography, Thoracic / Q000379:methods*; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D003074:Solitary Pulmonary Nodule / Q000000981:diagnostic imaging*