title: Estimating misclassification error: a closer look at cross-validation based methods.
abstract: BACKGROUND
  To estimate a classifier's error in predicting future observations, bootstrap methods have been proposed as reduced-variation alternatives to traditional cross-validation (CV) methods based on sampling without replacement. Monte Carlo (MC) simulation studies aimed at estimating the true misclassification error conditional on the training set are commonly used to compare CV methods. We conducted an MC simulation study to compare a new method of bootstrap CV (BCV) to k-fold CV for estimating clasification error.
  
  
  RESULTS
  For the low-dimensional conditions simulated, the modest positive bias of k-fold CV contrasted sharply with the substantial negative bias of the new BCV method. This behavior was corroborated using a real-world dataset of prognostic gene-expression profiles in breast cancer patients. Our simulation results demonstrate some extreme characteristics of variance and bias that can occur due to a fault in the design of CV exercises aimed at estimating the true conditional error of a classifier, and that appear not to have been fully appreciated in previous studies. Although CV is a sound practice for estimating a classifier's generalization error, using CV to estimate the fixed misclassification error of a trained classifier conditional on the training set is problematic. While MC simulation of this estimation exercise can correctly represent the average bias of a classifier, it will overstate the between-run variance of the bias.
  
  
  CONCLUSIONS
  We recommend k-fold CV over the new BCV method for estimating a classifier's generalization error. The extreme negative bias of BCV is too high a price to pay for its reduced variance.
authors.affiliation: Department of Biostatistics, University of Arkansas for Medical Sciences, 4301 W. Markham St. Slot 781, Little Rock, AR 72205, USA. STOunpraseuth@uams.edu
authors.forename: Songthip
authors.initials: S
authors.lastname: Ounpraseuth
authors.forename: Shelly Y
authors.initials: SY
authors.lastname: Lensing
authors.forename: Horace J
authors.initials: HJ
authors.lastname: Spencer
authors.forename: Ralph L
authors.initials: RL
authors.lastname: Kodell
mesh_terms: D000465:Algorithms; D001943:Breast Neoplasms / Q000175:diagnosis / Q000235:genetics*; D003198:Computer Simulation; D005260:Female; D015870:Gene Expression; D020869:Gene Expression Profiling; D006801:Humans; D009010:Monte Carlo Method; D020411:Oligonucleotide Array Sequence Analysis; D011237:Predictive Value of Tests; D011379:Prognosis; D012107:Research Design