title: Is mutual information adequate for feature selection in regression?
abstract: Feature selection is an important preprocessing step for many high-dimensional regression problems. One of the most common strategies is to select a relevant feature subset based on the mutual information criterion. However, no connection has been established yet between the use of mutual information and a regression error criterion in the machine learning literature. This is obviously an important lack, since minimising such a criterion is eventually the objective one is interested in. This paper demonstrates that under some reasonable assumptions, features selected with the mutual information criterion are the ones minimising the mean squared error and the mean absolute error. On the contrary, it is also shown that the mutual information criterion can fail in selecting optimal features in some situations that we characterise. The theoretical developments presented in this work are expected to lead in practice to a critical and efficient use of the mutual information for feature selection.
authors.affiliation: Machine Learning Group - ICTEAM, Université catholique de Louvain, Place du Levant 3, 1348 Louvain-la-Neuve, Belgium. Electronic address: benoit.frenay@uclouvain.be.
authors.forename: Benoît
authors.initials: B
authors.lastname: Frénay
authors.forename: Gauthier
authors.initials: G
authors.lastname: Doquire
authors.forename: Michel
authors.initials: M
authors.lastname: Verleysen
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D019277:Entropy; D007091:Image Processing, Computer-Assisted; D048088:Informatics; D016247:Information Storage and Retrieval; D016571:Neural Networks, Computer; D016011:Normal Distribution; D012044:Regression Analysis; D059629:Signal-To-Noise Ratio
keywords: Feature selection; MAE; MSE; Mutual information; Regression