Comparison of eight computer programs for receiver-operating characteristic analysis.
BACKGROUND
ROC analysis is widely accepted to assess and compare diagnostic validity of laboratory tests. Within the last few years, many new ROC programs have become available but have not been systematically evaluated. The aim of this study was to assess different ROC programs regarding their ease of use, mathematical correctness, final output, and their compatibility with other graphics programs.


METHODS
Eight available programs running under Windows (AccuROC, Analyse-It, CMDT, GraphROC, MedCalc, mROC, ROCKIT, and SPSS) were evaluated. ROC analyses of prostate-specific antigen and related values were performed from a dataset of 928 men with prostate cancer and benign prostatic hyperplasia and corresponding subsets. Criteria such as data input, data output, and correctness and completeness of results were used to evaluate the practicability of the programs.


RESULTS
Although the programs produced equivalent results (areas under the curves and their characteristics), we observed deficiencies concerning input of data, processing of the output data, and completeness of the results. Analyse-It, AccuROC, and MedCalc exhibited good performance, but each program had different shortcomings. Only GraphROC could compare curves at a certain sensitivity or specificity cutoff.


CONCLUSIONS
Adequate ROC analysis and ROC plotting cannot be performed with a single program. Analyse-It, AccuROC, and MedCalc can be recommended with certain limitations. Further improvements of the programs are necessary.
Stephan
Carsten
C
Department of Urology, University Hospital Charité, Humboldt University, D-10098 Berlin, Germany.
Wesseling
Sebastian
S
Schink
Tania
T
Jung
Klaus
K
D019540:Area Under Curve; D006801:Humans; D008297:Male; D016571:Neural Networks, Computer; D017430:Prostate-Specific Antigen / Q000032:analysis; D011470:Prostatic Hyperplasia / Q000175:diagnosis; D011471:Prostatic Neoplasms / Q000175:diagnosis; D012372:ROC Curve; D012984:Software
D017430:Prostate-Specific Antigen