title: Bounding the bias of contrastive divergence learning.
abstract: Optimization based on k-step contrastive divergence (CD) has become a common way to train restricted Boltzmann machines (RBMs). The k-step CD is a biased estimator of the log-likelihood gradient relying on Gibbs sampling. We derive a new upper bound for this bias. Its magnitude depends on k, the number of variables in the RBM, and the maximum change in energy that can be produced by changing a single variable. The last reflects the dependence on the absolute values of the RBM parameters. The magnitude of the bias is also affected by the distance in variation between the modeled distribution and the starting distribution of the Gibbs chain.
authors.affiliation: Institut für Neuroinformatik, Ruhr-Universität Bochum, 44780 Bochum, Germany asja.fischer@ini.rub.de.
authors.forename: Asja
authors.initials: A
authors.lastname: Fischer
authors.forename: Christian
authors.initials: C
authors.lastname: Igel
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D016013:Likelihood Functions; D008390:Markov Chains; D016571:Neural Networks, Computer