title: The feature selection bias problem in relation to high-dimensional gene data.
abstract: OBJECTIVE
  Feature selection is a technique widely used in data mining. The aim is to select the best subset of features relevant to the problem being considered. In this paper, we consider feature selection for the classification of gene datasets. Gene data is usually composed of just a few dozen objects described by thousands of features. For this kind of data, it is easy to find a model that fits the learning data. However, it is not easy to find one that will simultaneously evaluate new data equally well as learning data. This overfitting issue is well known as regards classification and regression, but it also applies to feature selection.
  
  
  METHODS
  We address this problem and investigate its importance in an empirical study of four feature selection methods applied to seven high-dimensional gene datasets. We chose datasets that are well studied in the literature-colon cancer, leukemia and breast cancer. All the datasets are characterized by a significant number of features and the presence of exactly two decision classes. The feature selection methods used are ReliefF, minimum redundancy maximum relevance, support vector machine-recursive feature elimination and relaxed linear separability.
  
  
  RESULTS
  Our main result reveals the existence of positive feature selection bias in all 28 experiments (7 datasets and 4 feature selection methods). Bias was calculated as the difference between validation and test accuracies and ranges from 2.6% to as much as 41.67%. The validation accuracy (biased accuracy) was calculated on the same dataset on which the feature selection was performed. The test accuracy was calculated for data that was not used for feature selection (by so called external cross-validation).
  
  
  CONCLUSIONS
  This work provides evidence that using the same dataset for feature selection and learning is not appropriate. We recommend using cross-validation for feature selection in order to reduce selection bias.
authors.affiliation: Faculty of Computer Science, Bialystok University of Technology, 45A Wiejska St., 15-351 Bialystok, Poland.
authors.forename: Jerzy
authors.initials: J
authors.lastname: Krawczuk
authors.affiliation: Faculty of Computer Science, Bialystok University of Technology, 45A Wiejska St., 15-351 Bialystok, Poland. Electronic address: t.lukaszuk@pb.edu.pl.
authors.forename: Tomasz
authors.initials: T
authors.lastname: Łukaszuk
mesh_terms: D000465:Algorithms; D015982:Bias; D014408:Biomarkers, Tumor / Q000235:genetics*; D019295:Computational Biology / Q000379:methods*; D057225:Data Mining / Q000379:methods*; D030541:Databases, Genetic; D003661:Decision Support Techniques; D020869:Gene Expression Profiling / Q000379:methods; D015972:Gene Expression Regulation, Neoplastic; D006801:Humans; D016014:Linear Models; D020411:Oligonucleotide Array Sequence Analysis; D010363:Pattern Recognition, Automated; D015203:Reproducibility of Results; D060388:Support Vector Machine
chemical_list: D014408:Biomarkers, Tumor
keywords: Convex and piecewise linear classifier; Feature selection bias; Gene selection; Microarray data; Support vector machine