Implementing online natural gradient learning: problems and solutions.
The online natural gradient learning is an efficient algorithm to resolve the slow learning speed and poor performance of the standard gradient descent method. However, there are several problems to implement this algorithm. In this paper, we proposed a new algorithm to solve these problems and then compared the new algorithm with other known algorithms for online learning, including Almeida-Langlois-Amaral-Plakhov algorithm (ALAP), Vario-eta, local adaptive learning rate and learning with momentum etc., using sample data sets from Proben1 and normalized handwritten digits, automatically scanned from envelopes by the U.S. Postal Services. The strong and weak points of these algorithms were analyzed and tested empirically. We found out that using the online training error as the criterion to determine whether the learning rate should be changed or not is not appropriate and our new algorithm has better performance than other existing online algorithms.
CED System Corporation, Jimbocho, Tokyo 101-0051, Japan. weisui@excite.co.jp
Weishui
W
Wan
D000465:Algorithms; D001185:Artificial Intelligence; D003198:Computer Simulation; D003661:Decision Support Techniques; D008962:Models, Theoretical; D016571:Neural Networks, Computer; D009716:Numerical Analysis, Computer-Assisted; D010363:Pattern Recognition, Automated / Q000379:methods*