title: A normalized adaptive training of recurrent neural networks with augmented error gradient.
abstract: For training algorithms of recurrent neural networks (RNN), convergent speed and training error are always two contradictory performances. In this letter, we propose a normalized adaptive recurrent learning (NARL) to obtain a tradeoff between transient and steady-state response. An augmented term is added to error gradient to exactly model the derivative of cost function with respect to hidden layer weight. The influence of the induced gain of activation function on training stability is also taken into consideration. Moreover, adaptive learning rate is employed to improve the robustness of the gradient training. Finally, computer simulations of a model prediction problem are synthesized to give comparisons between NARL and conventional normalized real-time recurrent learning (N-RTRL).
authors.affiliation: School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798, Singapore. bieber@pmail.ntu.edu.sg
authors.forename: W
authors.initials: W
authors.lastname: Yilei
authors.forename: S
authors.initials: S
authors.lastname: Qing
authors.forename: L
authors.initials: L
authors.lastname: Sheng
mesh_terms: D000220:Adaptation, Biological; D000465:Algorithms; D003198:Computer Simulation; D006801:Humans; D007858:Learning / Q000502:physiology*; D016571:Neural Networks, Computer; D012815:Signal Processing, Computer-Assisted; D013997:Time Factors