title: Efficient sample reuse in policy gradients with parameter-based exploration.
abstract: The policy gradient approach is a flexible and powerful reinforcement learning method particularly for problems with continuous actions such as robot control. A common challenge is how to reduce the variance of policy gradient estimates for reliable policy updates. In this letter, we combine the following three ideas and give a highly effective policy gradient method: (1) policy gradients with parameter-based exploration, a recently proposed policy search method with low variance of gradient estimates; (2) an importance sampling technique, which allows us to reuse previously gathered data in a consistent way; and (3) an optimal baseline, which minimizes the variance of gradient estimates with their unbiasedness being maintained. For the proposed method, we give a theoretical analysis of the variance of gradient estimates and show its usefulness through extensive experiments.
authors.affiliation: Department of Computer Science, Tokyo Institute of Technology, Tokyo 152-8552, Japan. tingting@sg.cs.titech.ac.jp
authors.forename: Tingting
authors.initials: T
authors.lastname: Zhao
authors.forename: Hirotaka
authors.initials: H
authors.lastname: Hachiya
authors.forename: Voot
authors.initials: V
authors.lastname: Tangkaratt
authors.forename: Jun
authors.initials: J
authors.lastname: Morimoto
authors.forename: Masashi
authors.initials: M
authors.lastname: Sugiyama
mesh_terms: D000465:Algorithms; D001132:Arm; D001185:Artificial Intelligence; D003198:Computer Simulation; D003657:Decision Making / Q000502:physiology*; D005106:Exploratory Behavior / Q000502:physiology*; D006801:Humans; D015233:Models, Statistical; D009068:Movement / Q000502:physiology; D017711:Nonlinear Dynamics; D012054:Reinforcement, Psychology