title: Cross-validation under separate sampling: strong bias and how to correct it.
abstract: BACKGROUND
  It is commonly assumed in pattern recognition that cross-validation error estimation is 'almost unbiased' as long as the number of folds is not too small. While this is true for random sampling, it is not true with separate sampling, where the populations are independently sampled, which is a common situation in bioinformatics.
  
  
  RESULTS
  We demonstrate, via analytical and numerical methods, that classical cross-validation can have strong bias under separate sampling, depending on the difference between the sampling ratios and the true population probabilities. We propose a new separate-sampling cross-validation error estimator, and prove that it satisfies an 'almost unbiased' theorem similar to that of random-sampling cross-validation. We present two case studies with previously published data, which show that the results can change drastically if the correct form of cross-validation is used.
  
  
  METHODS
  The source code in C++, along with the Supplementary Materials, is available at: http://gsp.tamu.edu/Publications/supplementary/zollanvari13/.
authors.affiliation: Department of Electrical and Computer Engineering, Center for Bioinformatics and Genomic Systems Engineering and Department of Statistics, Texas A&M University, College Station, TX, 77843, USA Department of Electrical and Computer Engineering, Center for Bioinformatics and Genomic Systems Engineering and Department of Statistics, Texas A&M University, College Station, TX, 77843, USA.
authors.forename: Ulisses M
authors.initials: UM
authors.lastname: Braga-Neto
authors.affiliation: Department of Electrical and Computer Engineering, Center for Bioinformatics and Genomic Systems Engineering and Department of Statistics, Texas A&M University, College Station, TX, 77843, USA Department of Electrical and Computer Engineering, Center for Bioinformatics and Genomic Systems Engineering and Department of Statistics, Texas A&M University, College Station, TX, 77843, USA.
authors.forename: Amin
authors.initials: A
authors.lastname: Zollanvari
authors.affiliation: Department of Electrical and Computer Engineering, Center for Bioinformatics and Genomic Systems Engineering and Department of Statistics, Texas A&M University, College Station, TX, 77843, USA Department of Electrical and Computer Engineering, Center for Bioinformatics and Genomic Systems Engineering and Department of Statistics, Texas A&M University, College Station, TX, 77843, USA.
authors.forename: Edward R
authors.initials: ER
authors.lastname: Dougherty
mesh_terms: D006801:Humans; D009369:Neoplasms / Q000235:genetics; D020411:Oligonucleotide Array Sequence Analysis; D010300:Parkinson Disease / Q000235:genetics; D011336:Probability; D015983:Selection Bias; D059467:Transcriptome