title: Bias correction for selecting the minimal-error classifier from many machine learning models.
abstract: BACKGROUND
  Supervised machine learning is commonly applied in genomic research to construct a classifier from the training data that is generalizable to predict independent testing data. When test datasets are not available, cross-validation is commonly used to estimate the error rate. Many machine learning methods are available, and it is well known that no universally best method exists in general. It has been a common practice to apply many machine learning methods and report the method that produces the smallest cross-validation error rate. Theoretically, such a procedure produces a selection bias. Consequently, many clinical studies with moderate sample sizes (e.g. n = 30-60) risk reporting a falsely small cross-validation error rate that could not be validated later in independent cohorts.
  
  
  RESULTS
  In this article, we illustrated the probabilistic framework of the problem and explored the statistical and asymptotic properties. We proposed a new bias correction method based on learning curve fitting by inverse power law (IPL) and compared it with three existing methods: nested cross-validation, weighted mean correction and Tibshirani-Tibshirani procedure. All methods were compared in simulation datasets, five moderate size real datasets and two large breast cancer datasets. The result showed that IPL outperforms the other methods in bias correction with smaller variance, and it has an additional advantage to extrapolate error estimates for larger sample sizes, a practical feature to recommend whether more samples should be recruited to improve the classifier and accuracy. An R package 'MLbias' and all source files are publicly available.
  
  
  METHODS
  tsenglab.biostat.pitt.edu/software.htm.
  
  
  BACKGROUND
  ctseng@pitt.edu
  
  
  BACKGROUND
  Supplementary data are available at Bioinformatics online.
authors.affiliation: Joint Carnegie Mellon University-University of Pittsburgh Ph.D. Program in Computational Biology, Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and Magee-Womens Research Institute, Pittsburgh, PA 15213, USA Joint Carnegie Mellon University-University of Pittsburgh Ph.D. Program in Computational Biology, Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and Magee-Womens Research Institute, Pittsburgh, PA 15213, USA.
authors.forename: Ying
authors.initials: Y
authors.lastname: Ding
authors.affiliation: Joint Carnegie Mellon University-University of Pittsburgh Ph.D. Program in Computational Biology, Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and Magee-Womens Research Institute, Pittsburgh, PA 15213, USA.
authors.forename: Shaowu
authors.initials: S
authors.lastname: Tang
authors.affiliation: Joint Carnegie Mellon University-University of Pittsburgh Ph.D. Program in Computational Biology, Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and Magee-Womens Research Institute, Pittsburgh, PA 15213, USA.
authors.forename: Serena G
authors.initials: SG
authors.lastname: Liao
authors.affiliation: Joint Carnegie Mellon University-University of Pittsburgh Ph.D. Program in Computational Biology, Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and Magee-Womens Research Institute, Pittsburgh, PA 15213, USA.
authors.forename: Jia
authors.initials: J
authors.lastname: Jia
authors.affiliation: Joint Carnegie Mellon University-University of Pittsburgh Ph.D. Program in Computational Biology, Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and Magee-Womens Research Institute, Pittsburgh, PA 15213, USA.
authors.forename: Steffi
authors.initials: S
authors.lastname: Oesterreich
authors.affiliation: Joint Carnegie Mellon University-University of Pittsburgh Ph.D. Program in Computational Biology, Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and Magee-Womens Research Institute, Pittsburgh, PA 15213, USA.
authors.forename: Yan
authors.initials: Y
authors.lastname: Lin
authors.affiliation: Joint Carnegie Mellon University-University of Pittsburgh Ph.D. Program in Computational Biology, Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and Magee-Womens Research Institute, Pittsburgh, PA 15213, USA Joint Carnegie Mellon University-University of Pittsburgh Ph.D. Program in Computational Biology, Department of Biostatistics, Graduate School of Public Health, University of Pittsburgh, Pittsburgh, PA 15261, USA and Magee-Womens Research Institute, Pittsburgh, PA 15213, USA.
authors.forename: George C
authors.initials: GC
authors.lastname: Tseng
mesh_terms: D001185:Artificial Intelligence; D001943:Breast Neoplasms / Q000235:genetics; D003627:Data Interpretation, Statistical; D005260:Female; D020869:Gene Expression Profiling; D023281:Genomics / Q000379:methods*; D006801:Humans; D008962:Models, Theoretical; D018401:Sample Size
references.citation: BMC Bioinformatics. 2008 Oct 16;9:439
references.pmid: 18925941
references.citation: J Am Stat Assoc. 2009 Sep 1;104(487):1015-1028
references.pmid: 20333278
references.citation: J Comput Biol. 2003;10(2):119-42
references.pmid: 12804087
references.citation: J Natl Cancer Inst. 2007 Jan 17;99(2):147-57
references.pmid: 17227998
references.citation: Nat Rev Genet. 2006 Jan;7(1):55-65
references.pmid: 16369572
references.citation: BMC Bioinformatics. 2006 Feb 23;7:91
references.pmid: 16504092
references.citation: BMC Med Res Methodol. 2009 Dec 21;9:85
references.pmid: 20025773
references.citation: Bioinformatics. 2005 May 1;21(9):1979-86
references.pmid: 15691862
references.citation: Bioinformatics. 2007 Jun 1;23(11):1363-70
references.pmid: 17392326
references.citation: Biometrics. 2013 Sep;69(3):693-702
references.pmid: 23845182
references.citation: Bioinformatics. 2006 May 15;22(10):1245-50
references.pmid: 16500931
references.citation: Bioinformatics. 2010 Jan 1;26(1):68-76
references.pmid: 19846436
references.citation: Nature. 2012 Apr 18;486(7403):346-52
references.pmid: 22522925