An information-theoretic approach to curiosity-driven reinforcement learning.
We provide a fresh look at the problem of exploration in reinforcement learning, drawing on ideas from information theory. First, we show that Boltzmann-style exploration, one of the main exploration methods used in reinforcement learning, is optimal from an information-theoretic point of view, in that it optimally trades expected return for the coding cost of the policy. Second, we address the problem of curiosity-driven learning. We propose that, in addition to maximizing the expected return, a learner should choose a policy that also maximizes the learner's predictive power. This makes the world both interesting and exploitable. Optimal policies then have the form of Boltzmann-style exploration with a bonus, containing a novel exploration-exploitation trade-off which emerges naturally from the proposed optimization principle. Importantly, this exploration-exploitation trade-off persists in the optimal deterministic policy, i.e., when there is no exploration due to randomness. As a result, exploration is understood as an emerging behavior that optimizes information gain, rather than being modeled as pure randomization of action choices.
Information and Computer Sciences, University of Hawaii at Mānoa, Honolulu, HI 96822, USA. sstill@hawaii.edu
Susanne
S
Still
Doina
D
Precup
D000465:Algorithms; D000818:Animals; D005106:Exploratory Behavior; D006801:Humans; D007257:Information Theory; D007858:Learning