title: Convergence of batch gradient learning with smoothing regularization and adaptive momentum for neural networks.
abstract: This paper presents new theoretical results on the backpropagation algorithm with smoothing [Formula: see text] regularization and adaptive momentum for feedforward neural networks with a single hidden layer, i.e., we show that the gradient of error function goes to zero and the weight sequence goes to a fixed point as n (n is iteration steps) tends to infinity, respectively. Also, our results are more general since we do not require the error function to be quadratic or uniformly convex, and neuronal activation functions are relaxed. Moreover, compared with existed algorithms, our novel algorithm can get more sparse network structure, namely it forces weights to become smaller during the training and can eventually removed after the training, which means that it can simply the network structure and lower operation time. Finally, two numerical experiments are presented to show the characteristics of the main results in detail.
authors.affiliation: School of Science, Xi'an Polytechnic University, Xi'an, 710048 People's Republic of China ; School of Mathematical Sciences, Dalian University of Technology, Dalian, 116024 People's Republic of China ; Department of Electrical and Computer Engineering, University of Louisville, Louisville, KY 40292 USA.
authors.forename: Qinwei
authors.initials: Q
authors.lastname: Fan
authors.affiliation: School of Mathematical Sciences, Dalian University of Technology, Dalian, 116024 People's Republic of China.
authors.forename: Wei
authors.initials: W
authors.lastname: Wu
authors.affiliation: Department of Electrical and Computer Engineering, University of Louisville, Louisville, KY 40292 USA ; Spoleczna Akademia Nauk, 90-011 Lodz, Poland.
authors.forename: Jacek M
authors.initials: JM
authors.lastname: Zurada
keywords: Adaptive momentum; Convergence; Feedforward neural networks; Smoothing [Formula: see text] regularization