title: Bias and Stability of Single Variable Classifiers for Feature Ranking and Selection.
abstract: Feature rankings are often used for supervised dimension reduction especially when discriminating power of each feature is of interest, dimensionality of dataset is extremely high, or computational power is limited to perform more complicated methods. In practice, it is recommended to start dimension reduction via simple methods such as feature rankings before applying more complex approaches. Single Variable Classifier (SVC) ranking is a feature ranking based on the predictive performance of a classifier built using only a single feature. While benefiting from capabilities of classifiers, this ranking method is not as computationally intensive as wrappers. In this paper, we report the results of an extensive study on the bias and stability of such feature ranking method. We study whether the classifiers influence the SVC rankings or the discriminative power of features themselves has a dominant impact on the final rankings. We show the common intuition of using the same classifier for feature ranking and final classification does not always result in the best prediction performance. We then study if heterogeneous classifiers ensemble approaches provide more unbiased rankings and if they improve final classification performance. Furthermore, we calculate an empirical prediction performance loss for using the same classifier in SVC feature ranking and final classification from the optimal choices.
authors.affiliation: Medical Image Analysis Laboratory, Department of Radiology, Henry Ford Health System, Detroit, MI 48202, USA ; Department of Computer Science, University of Maryland, College Park, MD 20740, USA.
authors.forename: Shobeir
authors.initials: S
authors.lastname: Fakhraei
authors.affiliation: Medical Image Analysis Laboratory, Department of Radiology, Henry Ford Health System, Detroit, MI 48202, USA ; Control and Intelligent Processing Center of Excellence (CIPCE), School of Electrical and Computer Engineering, University of Tehran, Tehran 14395, Iran.
authors.forename: Hamid
authors.initials: H
authors.lastname: Soltanian-Zadeh
authors.affiliation: College of Engineering, Wayne State University, Detroit, MI 48202, USA.
authors.forename: Farshad
authors.initials: F
authors.lastname: Fotouhi
keywords: AdaBoost; Bias; Dimension Reduction; Feature Ranking; Feature Selection; K-Nearest Neighbors; Logistic Regression; Multilayer Perceptron; Naïve Bayes; Random Forests; Single Variable Classifier; Stability; Support Vector Machines