title: How good is crude MDL for solving the bias-variance dilemma? An empirical investigation based on Bayesian networks.
abstract: The bias-variance dilemma is a well-known and important problem in Machine Learning. It basically relates the generalization capability (goodness of fit) of a learning method to its corresponding complexity. When we have enough data at hand, it is possible to use these data in such a way so as to minimize overfitting (the risk of selecting a complex model that generalizes poorly). Unfortunately, there are many situations where we simply do not have this required amount of data. Thus, we need to find methods capable of efficiently exploiting the available data while avoiding overfitting. Different metrics have been proposed to achieve this goal: the Minimum Description Length principle (MDL), Akaike's Information Criterion (AIC) and Bayesian Information Criterion (BIC), among others. In this paper, we focus on crude MDL and empirically evaluate its performance in selecting models with a good balance between goodness of fit and complexity: the so-called bias-variance dilemma, decomposition or tradeoff. Although the graphical interaction between these dimensions (bias and variance) is ubiquitous in the Machine Learning literature, few works present experimental evidence to recover such interaction. In our experiments, we argue that the resulting graphs allow us to gain insights that are difficult to unveil otherwise: that crude MDL naturally selects balanced models in terms of bias-variance, which not necessarily need be the gold-standard ones. We carry out these experiments using a specific model: a Bayesian network. In spite of these motivating results, we also should not overlook three other components that may significantly affect the final model selection: the search procedure, the noise rate and the sample size.
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Nicandro
authors.initials: N
authors.lastname: Cruz-Ramírez
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Héctor Gabriel
authors.initials: HG
authors.lastname: Acosta-Mesa
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Efrén
authors.initials: E
authors.lastname: Mezura-Montes
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Alejandro
authors.initials: A
authors.lastname: Guerra-Hernández
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Guillermo de Jesús
authors.initials: Gde J
authors.lastname: Hoyos-Rivera
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Rocío Erandi
authors.initials: RE
authors.lastname: Barrientos-Martínez
authors.affiliation: Centro de Investigaciones Biomédicas, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Karina
authors.initials: K
authors.lastname: Gutiérrez-Fragoso
authors.affiliation: Centro de Alta Tecnología de Educación a Distancia UNAM, Tlaxcala, Tlaxcala, México.
authors.forename: Luis Alonso
authors.initials: LA
authors.lastname: Nava-Fernández
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Patricia
authors.initials: P
authors.lastname: González-Gaspar
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Elva María
authors.initials: EM
authors.lastname: Novoa-del-Toro
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: Vicente Josué
authors.initials: VJ
authors.lastname: Aguilera-Rueda
authors.affiliation: Facultad de Física e Inteligencia Artificial, Universidad Veracruzana, Xalapa, Veracruz, México.
authors.forename: María Yaneli
authors.initials: MY
authors.lastname: Ameca-Alducin
mesh_terms: D000465:Algorithms; D001499:Bayes Theorem; D015982:Bias; D019992:Databases as Topic; D011336:Probability