Fast curvature matrix-vector products for second-order gradient descent.
We propose a generic method for iteratively approximating various second-order gradient steps - Newton, Gauss-Newton, Levenberg-Marquardt, and natural gradient - in linear time per iteration, using special curvature matrix-vector products that can be computed in O(n). Two recent acceleration techniques for on-line learning, matrix momentum and stochastic meta-descent (SMD), implement this approach. Since both were originally derived by very different routes, this offers fresh insight into their operation, resulting in further improvements to SMD.
Schraudolph
Nicol N
NN
IDSIA, Galleria 2, 6928 Manno, Switzerland. nic@inf.ethz.ch
D000054:Acceleration; D000465:Algorithms; D016571:Neural Networks, Computer; D013269:Stochastic Processes