title: Batch gradient method with smoothing L1/2 regularization for training of feedforward neural networks.
abstract: The aim of this paper is to develop a novel method to prune feedforward neural networks by introducing an L1/2 regularization term into the error function. This procedure forces weights to become smaller during the training and can eventually removed after the training. The usual L1/2 regularization term involves absolute values and is not differentiable at the origin, which typically causes oscillation of the gradient of the error function during the training. A key point of this paper is to modify the usual L1/2 regularization term by smoothing it at the origin. This approach offers the following three advantages: First, it removes the oscillation of the gradient value. Secondly, it gives better pruning, namely the final weights to be removed are smaller than those produced through the usual L1/2 regularization. Thirdly, it makes it possible to prove the convergence of the training. Supporting numerical examples are also provided.
authors.affiliation: School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, PR China.
authors.forename: Wei
authors.initials: W
authors.lastname: Wu
authors.affiliation: School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, PR China; Department of Electrical and Computer Engineering, University of Louisville, Louisville, KY 40292, USA.
authors.forename: Qinwei
authors.initials: Q
authors.lastname: Fan
authors.affiliation: Department of Electrical and Computer Engineering, University of Louisville, Louisville, KY 40292, USA; Spoleczna Akademia Nauk, 90-011 Lodz, Poland. Electronic address: jmzura02@louisville.edu.
authors.forename: Jacek M
authors.initials: JM
authors.lastname: Zurada
authors.affiliation: School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, PR China; School of Mathematics and Computational Sciences, China University of Petroleum, Dongying 257061, PR China.
authors.forename: Jian
authors.initials: J
authors.lastname: Wang
authors.affiliation: School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, PR China.
authors.forename: Dakun
authors.initials: D
authors.lastname: Yang
authors.affiliation: School of Mathematical Sciences, Dalian University of Technology, Dalian 116024, PR China; School of information Science and Engineering, Dalian Polytechnic University, Dalian 116034, PR China.
authors.forename: Yan
authors.initials: Y
authors.lastname: Liu
mesh_terms: D000465:Algorithms; D003198:Computer Simulation; D005246:Feedback; D006801:Humans; D007858:Learning; D016571:Neural Networks, Computer
keywords: Batch gradient method; Convergence; Feedforward neural networks; Smoothing  regularization