Independent versus sequential reading in ROC studies of computer-assist modalities: analysis of components of variance.
OBJECTIVE
The authors analyzed two methods for arranging the temporal sequencing of unaided versus computer-assisted reading in multiple-reader, multiple-case receiver operating characteristic studies of detection of solitary pulmonary nodules on chest radiographs. In the "independent" mode the readings are separated by about I month; in the "sequential" mode, the computer-assisted reading immediately follows the unassisted reading.


METHODS
The authors used the method of Beiden, Wagner, and Campbell (BWC) to decompose the components of variance of receiver operating characteristic accuracy measures into those that are correlated and those that are uncorrelated across reading conditions. Only the latter contribute to uncertainty in estimates of the difference in accuracy measures across reading conditions (unaided vs aided). This method was used to analyze data from two independent studies of the detection of solitary pulmonary nodules on chest radiographs.


RESULTS
In the sequential reading mode the components that were correlated across reading conditions increased compared to the independent reading mode, as might be expected. What was not anticipated was the fact that the total reader variance was approximately the same for the two reading modes. The results were remarkably similar across the two independent studies analyzed.


CONCLUSIONS
The sequential reading mode may thus be the more sensitive probe of the difference between unassisted and computer-assisted reading, if the mean effect is unperturbed (as here). It is also the least demanding on the logistics and investment of reader time.
Beiden
Sergey V
SV
Center for Devices and Radiological Health, Food and Drug Administration, Rockville, MD 20857, USA.
Wagner
Robert F
RF
Doi
Kunio
K
Nishikawa
Robert M
RM
Freedman
Matthew
M
Lo
Shih-Chung Ben
SC
Xu
Xin-Wei
XW
D000704:Analysis of Variance; D003936:Diagnosis, Computer-Assisted; D006801:Humans; D012372:ROC Curve; D013902:Radiography, Thoracic; D003074:Solitary Pulmonary Nodule / Q000000981:diagnostic imaging*