title: Parameter-exploring policy gradients.
abstract: We present a model-free reinforcement learning method for partially observable Markov decision problems. Our method estimates a likelihood gradient by sampling directly in parameter space, which leads to lower variance gradient estimates than obtained by regular policy gradient methods. We show that for several complex control tasks, including robust standing with a humanoid robot, this method outperforms well-known algorithms from the fields of standard policy gradients, finite difference methods and population based heuristics. We also show that the improvement is largest when the parameter samples are drawn symmetrically. Lastly we analyse the importance of the individual components of our method by incrementally incorporating them into the other algorithms, and measuring the gain in performance after each step.
authors.affiliation: Faculty of Computer Science, Technische Universität München, Boltzmannstr.3, 85748 Garching, Germany. sehnke@in.tum.de
authors.forename: Frank
authors.initials: F
authors.lastname: Sehnke
authors.forename: Christian
authors.initials: C
authors.lastname: Osendorfer
authors.forename: Thomas
authors.initials: T
authors.lastname: Rückstiess
authors.forename: Alex
authors.initials: A
authors.lastname: Graves
authors.forename: Jan
authors.initials: J
authors.lastname: Peters
authors.forename: Jürgen
authors.initials: J
authors.lastname: Schmidhuber
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D003198:Computer Simulation; D003661:Decision Support Techniques; D008390:Markov Chains; D015233:Models, Statistical; D016571:Neural Networks, Computer; D012054:Reinforcement, Psychology; D012371:Robotics