Adaptive importance sampling to accelerate training of a neural probabilistic language model.
Previous work on statistical language modeling has shown that it is possible to train a feedforward neural network to approximate probabilities over sequences of words, resulting in significant error reduction when compared to standard baseline models based on n-grams. However, training the neural network model with the maximum-likelihood criterion requires computations proportional to the number of words in the vocabulary. In this paper, we introduce adaptive importance sampling as a way to accelerate training of the model. The idea is to use an adaptive n-gram model to track the conditional distributions produced by the neural network. We show that a very significant speedup can be obtained on standard problems.
Bengio
Y
Y
Department of IRO, Universite de Montreal, Montreal, Canada. bengioy@iro.umontreal.ca
Senecal
J S
JS
D003198:Computer Simulation; D006801:Humans; D007802:Language; D008390:Markov Chains; D015233:Models, Statistical; D016571:Neural Networks, Computer; D011381:Programming Languages