Steepest descent with momentum for quadratic functions is a version of the conjugate gradient method.
It is pointed out that the so called momentum method, much used in the neural network literature as an acceleration of the backpropagation method, is a stationary version of the conjugate gradient method. Connections with the continuous optimization method known as heavy ball with friction are also made. In both cases, adaptive (dynamic) choices of the so called learning rate and momentum parameters are obtained using a control Liapunov function analysis of the system.
Bhaya
Amit
A
Department of Electrical Engineering, Federal University of Rio de Janeiro, PEE/COPPE/UFRJ, PO Box 68504, Rio de Janeiro, RJ 21945-970, Brazil. amit@nacad.ufrj.br
Kaszkurewicz
Eugenius
E
D000465:Algorithms; D001185:Artificial Intelligence; D003198:Computer Simulation; D005246:Feedback; D008959:Models, Neurological; D016571:Neural Networks, Computer; D017711:Nonlinear Dynamics