Reader studies for validation of CAD systems.
Evaluation of computational intelligence (CI) systems designed to improve the performance of a human operator is complicated by the need to include the effect of human variability. In this paper we consider human (reader) variability in the context of medical imaging computer-assisted diagnosis (CAD) systems, and we outline how to compare the detection performance of readers with and without the CAD. An effective and statistically powerful comparison can be accomplished with a receiver operating characteristic (ROC) experiment, summarized by the reader-averaged area under the ROC curve (AUC). The comparison requires sophisticated yet well-developed methods for multi-reader multi-case (MRMC) variance analysis. MRMC variance analysis accounts for random readers, random cases, and correlations in the experiment. In this paper, we extend the methods available for estimating this variability. Specifically, we present a method that can treat arbitrary study designs. Most methods treat only the fully-crossed study design, where every reader reads every case in two experimental conditions. We demonstrate our method with a computer simulation, and we assess the statistical power of a variety of study designs.
Gallas
Brandon D
BD
NIBIB/CDRH Laboratory for the Assessment of Medical Imaging Systems, FDA, Silver Spring, MD 20993-0002, United States. brandon.gallas@fda.hhs.gov
Brown
David G
DG
D035843:Biomedical Research; D003198:Computer Simulation; D003936:Diagnosis, Computer-Assisted; D003952:Diagnostic Imaging; D004194:Disease; D006801:Humans; D009010:Monte Carlo Method; D016571:Neural Networks, Computer; D012372:ROC Curve; D015203:Reproducibility of Results; D013045:Species Specificity