title: Statistically controlled activation weight initialization (SCAWI).
abstract: An optimum weight initialization which strongly improves the performance of the back propagation (BP) algorithm is suggested. By statistical analysis, the scale factor, R (which is proportional to the maximum magnitude of the weights), is obtained as a function of the paralyzed neuron percentage (PNP). Also, by computer simulation, the performances on the convergence speed have been related to PNP. An optimum range for R is shown to exist in order to minimize the time needed to reach the minimum of the cost function. Normalization factors are properly defined, which leads to a distribution of the activations independent of the neurons, and to a single nondimensional quantity, R, the value of which can be quickly found by computer simulation.
authors.affiliation: Istituto per i Circuiti Elettronici, CNR, Genova.
authors.forename: G P
authors.initials: GP
authors.lastname: Drago
authors.forename: S
authors.initials: S
authors.lastname: Ridella