title: Noise-enhanced convolutional neural networks.
abstract: Injecting carefully chosen noise can speed convergence in the backpropagation training of a convolutional neural network (CNN). The Noisy CNN algorithm speeds training on average because the backpropagation algorithm is a special case of the generalized expectation-maximization (EM) algorithm and because such carefully chosen noise always speeds up the EM algorithm on average. The CNN framework gives a practical way to learn and recognize images because backpropagation scales with training data. It has only linear time complexity in the number of training samples. The Noisy CNN algorithm finds a special separating hyperplane in the network's noise space. The hyperplane arises from the likelihood-based positivity condition that noise-boosts the EM algorithm. The hyperplane cuts through a uniform-noise hypercube or Gaussian ball in the noise space depending on the type of noise used. Noise chosen from above the hyperplane speeds training on average. Noise chosen from below slows it on average. The algorithm can inject noise anywhere in the multilayered network. Adding noise to the output neurons reduced the average per-iteration training-set cross entropy by 39% on a standard MNIST image test set of handwritten digits. It also reduced the average per-iteration training-set classification error by 47%. Adding noise to the hidden layers can also reduce these performance measures. The noise benefit is most pronounced for smaller data sets because the largest EM hill-climbing gains tend to occur in the first few iterations. This noise effect can assist random sampling from large data sets because it allows a smaller random sample to give the same or better performance than a noiseless sample gives.
authors.affiliation: Signal and Information Processing Institute, Electrical Engineering Department, University of Southern California, Los Angeles, CA, United States.
authors.forename: Kartik
authors.initials: K
authors.lastname: Audhkhasi
authors.affiliation: Signal and Information Processing Institute, Electrical Engineering Department, University of Southern California, Los Angeles, CA, United States.
authors.forename: Osonde
authors.initials: O
authors.lastname: Osoba
authors.affiliation: Signal and Information Processing Institute, Electrical Engineering Department, University of Southern California, Los Angeles, CA, United States. Electronic address: kosko@sipi.usc.edu.
authors.forename: Bart
authors.initials: B
authors.lastname: Kosko
mesh_terms: D000465:Algorithms; D007858:Learning / Q000502:physiology; D016013:Likelihood Functions; D016571:Neural Networks, Computer; D016011:Normal Distribution; D011897:Random Allocation
keywords: Backpropagation; Convolutional neural network; Expectation–maximization algorithm; Noise injection; Sampling from big data sets; Stochastic resonance