title: On adaptive learning rate that guarantees convergence in feedforward networks.
abstract: This paper investigates new learning algorithms (LF I and LF II) based on Lyapunov function for the training of feedforward neural networks. It is observed that such algorithms have interesting parallel with the popular backpropagation (BP) algorithm where the fixed learning rate is replaced by an adaptive learning rate computed using convergence theorem based on Lyapunov stability theory. LF II, a modified version of LF I, has been introduced with an aim to avoid local minima. This modification also helps in improving the convergence speed in some cases. Conditions for achieving global minimum for these kind of algorithms have been studied in detail. The performances of the proposed algorithms are compared with BP algorithm and extended Kalman filtering (EKF) on three bench-mark function approximation problems: XOR, 3-bit parity, and 8-3 encoder. The comparisons are made in terms of number of learning iterations and computational time required for convergence. It is found that the proposed algorithms (LF I and II) are much faster in convergence than other two algorithms to attain same accuracy. Finally, the comparison is made on a complex two-dimensional (2-D) Gabor function and effect of adaptive learning rate for faster convergence is verified. In a nutshell, the investigations made in this paper help us better understand the learning procedure of feedforward neural networks in terms of adaptive learning rate, convergence speed, and local minima.
authors.affiliation: Department of Electrical Engineering, Indian Institute of Technology, Kanpur 208 016, India. lbehera@iitk.ac.in
authors.forename: Laxmidhar
authors.initials: L
authors.lastname: Behera
authors.forename: Swagat
authors.initials: S
authors.lastname: Kumar
authors.forename: Awhan
authors.initials: A
authors.lastname: Patnaik
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D016000:Cluster Analysis; D003205:Computing Methodologies; D005246:Feedback; D015233:Models, Statistical; D016571:Neural Networks, Computer; D010363:Pattern Recognition, Automated / Q000379:methods*; D013598:Systems Theory