title: A dynamical system perspective of structural learning with forgetting.
abstract: Structural learning with forgetting is an established method of using Laplace regularization to generate skeletal artificial neural networks. In this paper we develop a continuous dynamical system model of regularization in which the associated regularization parameter is generalized to be a time-varying function. Analytic results are obtained for a Laplace regularizer and a quadratic error surface by solving a different linear system in each region of the weight space. This model also enables a comparison of Laplace and Gaussian regularization. Both of these regularizers have a greater effect in weight space directions which are less important for minimization of a quadratic error function. However, for the Gaussian regularizer, the regularization parameter modifies the associated linear system eigenvalues, in contrast to its function as a control input in the Laplace case. This difference provides additional evidence for the superiority of the Laplace over the Gaussian regularizer.
authors.affiliation: Department of Electrical and Computer Engineering, Western Michigan University, Kalamazoo, MI 49008, USA.
authors.forename: D A
authors.initials: DA
authors.lastname: Miller
authors.forename: J M
authors.initials: JM
authors.lastname: Zurada