Generalized RLS approach to the training of neural networks.
Recursive least square (RLS) is an efficient approach to neural network training. However, in the classical RLS algorithm, there is no explicit decay in the energy function. This will lead to an unsatisfactory generalization ability for the trained networks. In this paper, we propose a generalized RLS (GRLS) model which includes a general decay term in the energy function for the training of feedforward neural networks. In particular, four different weight decay functions, namely, the quadratic weight decay, the constant weight decay and the newly proposed multimodal and quartic weight decay are discussed. By using the GRLS approach, not only the generalization ability of the trained networks is significantly improved but more unnecessary weights are pruned to obtain a compact network. Furthermore, the computational complexity of the GRLS remains the same as that of the standard RLS algorithm. The advantages and tradeoffs of using different decay functions are analyzed and then demonstrated with examples. Simulation results show that our approach is able to meet the design goals: improving the generalization ability of the trained network while getting a compact network.
Xu
Yong
Y
Centre of Excellence for Research in Computational Intelligence and Applications, School of Computer Science, The University of Birmingham, Edgbaston, Birmingham, B15 2TT, UK. y.xu@cs.bham.ac.uk
Wong
Kwok-Wo
KW
Leung
Chi-Sing
CS
D000465:Algorithms; D001185:Artificial Intelligence; D001943:Breast Neoplasms / Q000145:classification; D003198:Computer Simulation; D005260:Female; D005544:Forecasting; D006801:Humans; D016018:Least-Squares Analysis; D008959:Models, Neurological; D016571:Neural Networks, Computer; D019445:Solar Activity