On structure-exploiting trust-region regularized nonlinear least squares algorithms for neural-network learning.
This paper briefly introduces our numerical linear algebra approaches for solving structured nonlinear least squares problems arising from 'multiple-output' neural-network (NN) models. Our algorithms feature trust-region regularization, and exploit sparsity of either the 'block-angular' residual Jacobian matrix or the 'block-arrow' Gauss-Newton Hessian (or Fisher information matrix in statistical sense) depending on problem scale so as to render a large class of NN-learning algorithms 'efficient' in both memory and operation costs. Using a relatively large real-world nonlinear regression application, we shall explain algorithmic strengths and weaknesses, analyzing simulation results obtained by both direct and iterative trust-region algorithms with two distinct NN models: 'multilayer perceptrons' (MLP) and 'complementary mixtures of MLP-experts' (or neuro-fuzzy modular networks).
Mizutani
Eiji
E
Department of Computer Science, National Tsing Hua University, Hsinchu 300, Taiwan, ROC. eiji@wayne.cs.nthu.edu.tw
Demmel
James W
JW
D000465:Algorithms; D007858:Learning / Q000502:physiology; D016018:Least-Squares Analysis; D016571:Neural Networks, Computer; D017711:Nonlinear Dynamics