Incorporating utility-weights when comparing two diagnostic systems: a preliminary assessment.
OBJECTIVE
We sought to develop a new index that incorporates utility-weights when assessing the overall performance of a diagnostic system and to provide a statistical test for comparing two indices in a paired study design.


METHODS
The area under the receiver operating characteristic (ROC) curve (AUC) was used as the basis for constructing a new index. The index we propose represents a weighted average of class-specific AUCs each of which relates to a class of pairs of actually negative (normal) and actually positive (abnormal) cases with a specific predetermined utility (or clinical importance). For each pair of normal-abnormal cases, the utility is defined a priori and based on external (covariate) information. In the proposed approach utility-weights represent the relative importance (utility) of discriminating between different types of normal and abnormal cases (pairs of the same type are combined in the classes termed utility-classes). We also describe a simple nonparametric procedure for comparing the proposed indices as computed from paired data. Computer simulations were conducted to evaluate the behavior of the type I error of the proposed test in the simple albeit important instance of two utility-classes.


RESULTS
The new index provides an extension of the commonly used area under the ROC curve. It allows for incorporation of utility-weights into the analysis and reduces to the conventional AUC index when all assigned utility-weights are equal to unity. Computer simulations indicate that in the considered scenario of two utility-classes, the type I error of the proposed test is comparable to that of the conventional nonparametric test for equality of AUC indices.


CONCLUSIONS
The proposed index and the statistical test provide a practical approach of incorporating utilities when comparing diagnostic systems.
Bandos
Andriy I
AI
Department of Biostatistics, Graduate School of Public Health, Department of Radiology, Imaging, Suite 4200, Magee-Women's Hospital, University of Pittsburgh, Pittsburgh, PA 15216, USA.
Rockette
Howard E
HE
Gur
David
D
D000465:Algorithms; D003627:Data Interpretation, Statistical; D003952:Diagnostic Imaging / Q000379:methods*; D005240:Feasibility Studies; D007090:Image Interpretation, Computer-Assisted / Q000379:methods*; D010865:Pilot Projects; D012372:ROC Curve; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity