Complexity issues in natural gradient descent method for training multilayer perceptrons.
The natural gradient descent method is applied to train an n-m-1 multilayer perceptron. Based on an efficient scheme to represent the Fisher information matrix for an n-m-1 stochastic multilayer perceptron, a new algorithm is proposed to calculate the natural gradient without inverting the Fisher information matrix explicitly. When the input dimension n is much larger than the number of hidden neurons m, the time complexity of computing the natural gradient is O(n).
Yang
H H
HH
Oregon Graduate Institute, Computer Science Dept, Box 91000, Portland OR 97291, USA. hyang@cse.ogi.edu
Amari
S
S
D007858:Learning; D008959:Models, Neurological; D016571:Neural Networks, Computer; D009474:Neurons; D013269:Stochastic Processes