title: Efficient calculation of the Gauss-Newton approximation of the Hessian matrix in neural networks.
abstract: The Levenberg-Marquardt (LM) learning algorithm is a popular algorithm for training neural networks; however, for large neural networks, it becomes prohibitively expensive in terms of running time and memory requirements. The most time-critical step of the algorithm is the calculation of the Gauss-Newton matrix, which is formed by multiplying two large Jacobian matrices together. We propose a method that uses backpropagation to reduce the time of this matrix-matrix multiplication. This reduces the overall asymptotic running time of the LM algorithm by a factor of the order of the number of output nodes in the neural network.
authors.affiliation: Department of Computing, School of Informatics, City University London, London EC1V 0HB, U.K. michael.fairbank1@city.ac.uk
authors.forename: Michael
authors.initials: M
authors.lastname: Fairbank
authors.forename: Eduardo
authors.initials: E
authors.lastname: Alonso
mesh_terms: D000465:Algorithms; D016571:Neural Networks, Computer