title: Boundedness and convergence analysis of weight elimination for cyclic training of neural networks.
abstract: Weight elimination offers a simple and efficient improvement of training algorithm of feedforward neural networks. It is a general regularization technique in terms of the flexible scaling parameters. Actually, the weight elimination technique also contains the weight decay regularization for a large scaling parameter. Many applications of this technique and its improvements have been reported. However, there is little research concentrated on its convergence behavior. In this paper, we theoretically analyze the weight elimination for cyclic learning method and determine the conditions for the uniform boundedness of weight sequence, and weak and strong convergence. Based on the assumed network parameters, the optimal choice for the scaling parameter can also be determined. Moreover, two illustrative simulations have been done to support the theoretical explorations as well.
authors.affiliation: College of Science, China University of Petroleum, Qingdao, 266580, China; Department of Electrical and Computer Engineering, University of Louisville, Louisville, KY, 40292, USA. Electronic address: wangjiannl@upc.edu.cn.
authors.forename: Jian
authors.initials: J
authors.lastname: Wang
authors.affiliation: College of Computer & Communication Engineering, China University of Petroleum, Qingdao, 266580, China.
authors.forename: Zhenyun
authors.initials: Z
authors.lastname: Ye
authors.affiliation: College of Science, China University of Petroleum, Qingdao, 266580, China.
authors.forename: Weifeng
authors.initials: W
authors.lastname: Gao
authors.affiliation: Department of Electrical and Computer Engineering, University of Louisville, Louisville, KY, 40292, USA; Information Technology Institute, University of Social Sciences, Łódź 90-113, Poland. Electronic address: jacek.zurada@louisville.edu.
authors.forename: Jacek M
authors.initials: JM
authors.lastname: Zurada
mesh_terms: D000465:Algorithms; D000069550:Machine Learning; D016571:Neural Networks, Computer
keywords: Boundedness; Convergence; Neural networks; Weight decay; Weight elimination