title: From free energy to expected energy: Improving energy-based value function approximation in reinforcement learning.
abstract: Free-energy based reinforcement learning (FERL) was proposed for learning in high-dimensional state and action spaces. However, the FERL method does only really work well with binary, or close to binary, state input, where the number of active states is fewer than the number of non-active states. In the FERL method, the value function is approximated by the negative free energy of a restricted Boltzmann machine (RBM). In our earlier study, we demonstrated that the performance and the robustness of the FERL method can be improved by scaling the free energy by a constant that is related to the size of network. In this study, we propose that RBM function approximation can be further improved by approximating the value function by the negative expected energy (EERL), instead of the negative free energy, as well as being able to handle continuous state input. We validate our proposed method by demonstrating that EERL: (1) outperforms FERL, as well as standard neural network and linear function approximation, for three versions of a gridworld task with high-dimensional image state input; (2) achieves new state-of-the-art results in stochastic SZ-Tetris in both model-free and model-based learning settings; and (3) significantly outperforms FERL and standard neural network function approximation for a robot navigation task with raw and noisy RGB images as state input and a large number of actions.
authors.affiliation: Department of Brain Robot Interface, ATR Computational Neuroscience Laboratories, 2-2-2 Hikaridai, Seikacho, Soraku-gun, Kyoto 619-0288, Japan; Okinawa Institute of Science and Technology Graduate University, 1919-1 Tancha, Onna-son, Okinawa 904-0495, Japan. Electronic address: elfwing@atr.jp.
authors.forename: Stefan
authors.initials: S
authors.lastname: Elfwing
authors.affiliation: Department of Brain Robot Interface, ATR Computational Neuroscience Laboratories, 2-2-2 Hikaridai, Seikacho, Soraku-gun, Kyoto 619-0288, Japan; Okinawa Institute of Science and Technology Graduate University, 1919-1 Tancha, Onna-son, Okinawa 904-0495, Japan. Electronic address: uchibe@atr.jp.
authors.forename: Eiji
authors.initials: E
authors.lastname: Uchibe
authors.affiliation: Okinawa Institute of Science and Technology Graduate University, 1919-1 Tancha, Onna-son, Okinawa 904-0495, Japan. Electronic address: doya@oist.jp.
authors.forename: Kenji
authors.initials: K
authors.lastname: Doya
mesh_terms: D006801:Humans; D007858:Learning; D000069550:Machine Learning / Q000639:trends; D008962:Models, Theoretical; D016571:Neural Networks, Computer; D010775:Photic Stimulation / Q000379:methods*; D011897:Random Allocation; D012054:Reinforcement, Psychology
keywords: Expected energy; Function approximation; Reinforcement learning; Restricted Boltzmann machine; SZ-Tetris