Small-sample precision of ROC-related estimates.
BACKGROUND
The receiver operator characteristic (ROC) curves are commonly used in biomedical applications to judge the performance of a discriminant across varying decision thresholds. The estimated ROC curve depends on the true positive rate (TPR) and false positive rate (FPR), with the key metric being the area under the curve (AUC). With small samples these rates need to be estimated from the training data, so a natural question arises: How well do the estimates of the AUC, TPR and FPR compare with the true metrics?


RESULTS
Through a simulation study using data models and analysis of real microarray data, we show that (i) for small samples the root mean square differences of the estimated and true metrics are considerable; (ii) even for large samples, there is only weak correlation between the true and estimated metrics; and (iii) generally, there is weak regression of the true metric on the estimated metric. For classification rules, we consider linear discriminant analysis, linear support vector machine (SVM) and radial basis function SVM. For error estimation, we consider resubstitution, three kinds of cross-validation and bootstrap. Using resampling, we show the unreliability of some published ROC results.


BACKGROUND
Companion web site at http://compbio.tgen.org/paper_supp/ROC/roc.html


BACKGROUND
edward@mail.ece.tamu.edu.
Hanczar
Blaise
B
LIPADE, University Paris Descartes, Paris, France.
Hua
Jianping
J
Sima
Chao
C
Weinstein
John
J
Bittner
Michael
M
Dougherty
Edward R
ER
D000465:Algorithms; D005189:False Positive Reactions; D020411:Oligonucleotide Array Sequence Analysis; D010363:Pattern Recognition, Automated / Q000379:methods; D012372:ROC Curve