title: A fast learning algorithm for deep belief nets.
abstract: We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.
authors.affiliation: Department of Computer Science, University of Toronto, Canada. hinton@cs.toronto.edu
authors.forename: Geoffrey E
authors.initials: GE
authors.lastname: Hinton
authors.forename: Simon
authors.initials: S
authors.lastname: Osindero
authors.forename: Yee-Whye
authors.initials: YW
authors.lastname: Teh
mesh_terms: D000465:Algorithms; D000818:Animals; D006801:Humans; D007858:Learning / Q000502:physiology*; D016571:Neural Networks, Computer; D009474:Neurons / Q000502:physiology*