Approaches to Regularized Regression - A Comparison between Gradient Boosting and the Lasso.
BACKGROUND
Penalization and regularization techniques for statistical modeling have attracted increasing attention in biomedical research due to their advantages in the presence of high-dimensional data. A special focus lies on algorithms that incorporate automatic variable selection like the least absolute shrinkage operator (lasso) or statistical boosting techniques.


OBJECTIVE
Focusing on the linear regression framework, this article compares the two most-common techniques for this task, the lasso and gradient boosting, both from a methodological and a practical perspective.


METHODS
We describe these methods highlighting under which circumstances their results will coincide in low-dimensional settings. In addition, we carry out extensive simulation studies comparing the performance in settings with more predictors than observations and investigate multiple combinations of noise-to-signal ratio and number of true non-zero coeffcients. Finally, we examine the impact of different tuning methods on the results.


RESULTS
Both methods carry out penalization and variable selection for possibly highdimensional data, often resulting in very similar models. An advantage of the lasso is its faster run-time, a strength of the boosting concept is its modular nature, making it easy to extend to other regression settings.


CONCLUSIONS
Although following different strategies with respect to optimization and regularization, both methods imply similar constraints to the estimation problem leading to a comparable performance regarding prediction accuracy and variable selection in practice.
Tobias
T
Hepp
Matthias
M
Schmid
Olaf
O
Gefeller
Elisabeth
E
Waldmann
Andreas
A
Mayr
D000465:Algorithms; D003198:Computer Simulation; D008962:Models, Theoretical; D009716:Numerical Analysis, Computer-Assisted; D012044:Regression Analysis; D015203:Reproducibility of Results; D012815:Signal Processing, Computer-Assisted; D012984:Software
Penalization; boosting; high-dimensional data; lasso; regularization; variable selection