title: Adaptive importance sampling to accelerate training of a neural probabilistic language model.
abstract: Previous work on statistical language modeling has shown that it is possible to train a feedforward neural network to approximate probabilities over sequences of words, resulting in significant error reduction when compared to standard baseline models based on n-grams. However, training the neural network model with the maximum-likelihood criterion requires computations proportional to the number of words in the vocabulary. In this paper, we introduce adaptive importance sampling as a way to accelerate training of the model. The idea is to use an adaptive n-gram model to track the conditional distributions produced by the neural network. We show that a very significant speedup can be obtained on standard problems.
authors.affiliation: Department of IRO, Universite de Montreal, Montreal, Canada. bengioy@iro.umontreal.ca
authors.forename: Y
authors.initials: Y
authors.lastname: Bengio
authors.forename: J S
authors.initials: JS
authors.lastname: Senecal
mesh_terms: D003198:Computer Simulation; D006801:Humans; D007802:Language; D008390:Markov Chains; D015233:Models, Statistical; D016571:Neural Networks, Computer; D011381:Programming Languages