title: Many regression algorithms, one unified model: A review.
abstract: Regression is the process of learning relationships between inputs and continuous outputs from example data, which enables predictions for novel inputs. The history of regression is closely related to the history of artificial neural networks since the seminal work of Rosenblatt (1958). The aims of this paper are to provide an overview of many regression algorithms, and to demonstrate how the function representation whose parameters they regress fall into two classes: a weighted sum of basis functions, or a mixture of linear models. Furthermore, we show that the former is a special case of the latter. Our ambition is thus to provide a deep understanding of the relationship between these algorithms, that, despite being derived from very different principles, use a function representation that can be captured within one unified model. Finally, step-by-step derivations of the algorithms from first principles and visualizations of their inner workings allow this article to be used as a tutorial for those new to regression.
authors.affiliation: Unité d'Informatique et d'Ingénierie des Systèmes, ENSTA ParisTech, Université Paris-Saclay, 828 bd des Maréchaux, 91762 Palaiseau cedex, France; FLOWERS Research Team, INRIA, Bordeaux, France. Electronic address: freek.stulp@ensta-paristech.fr.
authors.forename: Freek
authors.initials: F
authors.lastname: Stulp
authors.affiliation: Sorbonne Universités, UPMC Univ Paris 06, UMR 7222, F-75005 Paris, France. Electronic address: olivier.sigaud@isir.upmc.fr.
authors.forename: Olivier
authors.initials: O
authors.lastname: Sigaud
mesh_terms: D000465:Algorithms; D006801:Humans; D016014:Linear Models; D008962:Models, Theoretical; D016571:Neural Networks, Computer; D016011:Normal Distribution
keywords: Gaussian mixture regression; Gaussian process regression; Locally weighted regression; Radial basis function networks; Regression