On 'natural' learning and pruning in multi-layered perceptrons.
Several studies have shown that natural gradient descent for on-line learning is much more efficient than standard gradient descent. In this article, we derive natural gradients in a slightly different manner and discuss implications for batch-mode learning and pruning, linking them to existing algorithms such as Levenberg-Marquardt optimization and optimal brain surgeon. The Fisher matrix plays an important role in all these algorithms. The second half of the article discusses a layered approximation of the Fisher matrix specific to multilayered perceptrons. Using this approximation rather than the exact Fisher matrix, we arrive at much faster "natural" learning algorithms and more robust pruning procedures.
Heskes
T
T
Foundation for Neural Networks, University of Nijmegen, 6525 EZ Nijmegen, The Netherlands.
D000465:Algorithms; D001185:Artificial Intelligence; D003663:Decision Trees; D016571:Neural Networks, Computer; D009493:Neurosurgery