What's the control in studies measuring the effect of computer-aided detection (CAD) on observer performance?
OBJECTIVE
The goal of many multiple-observer computer-aided detection (CADe) studies is to estimate the change in observers' diagnostic performance with CADe from their unaided performance. A key issue in these studies is the method for estimating the observers' unaided performance. The crossover design is considered the most valid. The sequential design takes less time and is less expensive but may be biased. We conducted a study to investigate the differences between these two designs.


METHODS
Data from two large CADe studies using both types of unaided reads were analyzed. The first study involved three radiologists examining the chest x-rays of 200 patients for lung nodules. The second study involved 19 observers interpreting the computed tomography colonography images of 100 patients for polyps. Observers' sensitivity, specificity, and receiver operating characteristic areas were estimated while unaided in both designs and compared to their accuracy with CADe. Bias, inter-observer variability, and correlations between unaided and aided results were assessed.


RESULTS
Observers tend to perform better while unaided in the sequential design than while unaided in the crossover design, but the differences are small. The inter-observer variability is larger in the sequential design. The correlations between unaided and aided results are larger in the sequential design. 95% CIs for the change with CADe are narrower with the sequential design.


CONCLUSIONS
The estimated effect of CADe on observer performance is similar regardless of the study design. Use of the sequential design may save investigators time and resources.
Obuchowski
Nancy A
NA
Quantitative Health Sciences, Cleveland Clinic Foundation, 9500 Euclid Avenue, Cleveland, OH 44195, USA. nobuchow@bio.ri.ccf.org
Meziane
Moulay
M
Dachman
Abraham H
AH
Lieber
Michael L
ML
Mazzone
Peter J
PJ
D000465:Algorithms; D018592:Cross-Over Studies; D007089:Image Enhancement / Q000379:methods; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D015588:Observer Variation; D010363:Pattern Recognition, Automated / Q000379:methods*; D011361:Professional Competence; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D014481:United States