Reader variance in ROC studies--generalizability to reader population at high and low performance levels.
OBJECTIVE
To investigate the variability between discriminative performances of readers as a function of average performance levels during receiver operating characteristic (ROC) studies.


METHODS
Four subsets of cases from previously ascertained ROC rating data by 12 observers when detecting interstitial disease and pneumothorax on posteroanterior chest films were selected for each abnormality and reanalyzed to assess changes in "reader" variance component. The subsets were selected based on a prestudy subjective assessment of the subtleness of depicted abnormality (positive cases) and the difficulty in determining its absence (negative cases). Reader variance component was estimated using a bootstrap approach for each subset and the results were used to assess a general relationship between variability and average performance level.


RESULTS
The reader variance component decreased substantially (from 0.007704 to 0.000426), as expected, when the areas under the ROC curves (AUC) for detecting pneumothoraces increased from 84% to 97%. On the other hand, reader variance component increased substantially (from 0.000890 to 0.005181) when AUC for detecting interstitial disease increased from 59% to 87%. The large magnitude of and changes in the reader variance component resulted in a consistent nonmonotone relationship as a function of AUC when other related variance components were included in addition to the reader component.


CONCLUSIONS
Among several factors affecting generalizability of ROC results to the population of readers, the reader variance component depended nonmonotonically on the average diagnostic performance and is lowest at both very high and very low levels of performance.
Gur
David
D
Department of Radiology, School of Medicine, University of Pittsburgh, Pittsburgh, PA 15261, USA. gurd@upmc.edu
Bandos
Andriy I
AI
Klym
Amy H
AH
Rockette
Howard E
HE
D002983:Clinical Competence; D006801:Humans; D017563:Lung Diseases, Interstitial / Q000000981:diagnostic imaging*; D015588:Observer Variation; D011030:Pneumothorax / Q000000981:diagnostic imaging*; D012372:ROC Curve; D011859:Radiography