title: Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method.
abstract: It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system.
authors.affiliation: Department of Electrical Engineering, Federal University of Rio de Janeiro, PEE/COPPE/UFRJ, PO Box 68504, Rio de Janeiro, RJ 21945-970, Brazil. amit@nacad.ufrj.br
authors.forename: Amit
authors.initials: A
authors.lastname: Bhaya
authors.forename: Eugenius
authors.initials: E
authors.lastname: Kaszkurewicz
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D003198:Computer Simulation; D005246:Feedback; D008959:Models, Neurological; D016571:Neural Networks, Computer; D017711:Nonlinear Dynamics