Improved variance estimation of classification performance via reduction of bias caused by small sample size.
BACKGROUND
Supervised learning for classification of cancer employs a set of design examples to learn how to discriminate between tumors. In practice it is crucial to confirm that the classifier is robust with good generalization performance to new examples, or at least that it performs better than random guessing. A suggested alternative is to obtain a confidence interval of the error rate using repeated design and test sets selected from available examples. However, it is known that even in the ideal situation of repeated designs and tests with completely novel samples in each cycle, a small test set size leads to a large bias in the estimate of the true variance between design sets. Therefore different methods for small sample performance estimation such as a recently proposed procedure called Repeated Random Sampling (RSS) is also expected to result in heavily biased estimates, which in turn translates into biased confidence intervals. Here we explore such biases and develop a refined algorithm called Repeated Independent Design and Test (RIDT).


RESULTS
Our simulations reveal that repeated designs and tests based on resampling in a fixed bag of samples yield a biased variance estimate. We also demonstrate that it is possible to obtain an improved variance estimate by means of a procedure that explicitly models how this bias depends on the number of samples used for testing. For the special case of repeated designs and tests using new samples for each design and test, we present an exact analytical expression for how the expected value of the bias decreases with the size of the test set.


CONCLUSIONS
We show that via modeling and subsequent reduction of the small sample bias, it is possible to obtain an improved estimate of the variance of classifier performance between design sets. However, the uncertainty of the variance estimate is large in the simulations performed indicating that the method in its present form cannot be directly applied to small data sets.
Department of Genetics and Pathology, Uppsala University, Rudbeck Laboratory, SE-751 85 Uppsala, Sweden. bolin@genpat.uu.se
Ulrika
U
Wickenberg-Bolin
Hanna
H
Göransson
Mårten
M
Fryknäs
Mats G
MG
Gustafsson
Anders
A
Isaksson
D000704:Analysis of Variance; D001185:Artificial Intelligence; D015982:Bias; D014408:Biomarkers, Tumor / Q000032:analysis*; D003936:Diagnosis, Computer-Assisted / Q000379:methods*; D020869:Gene Expression Profiling / Q000379:methods*; D006801:Humans; D008954:Models, Biological; D015233:Models, Statistical; D009363:Neoplasm Proteins / Q000032:analysis*; D009369:Neoplasms / Q000175:diagnosis* / Q000378:metabolism; D020411:Oligonucleotide Array Sequence Analysis / Q000379:methods; D010363:Pattern Recognition, Automated / Q000379:methods; D015203:Reproducibility of Results; D018401:Sample Size; D012680:Sensitivity and Specificity
D014408:Biomarkers, Tumor; D009363:Neoplasm Proteins