title: Human and machine learning in non-Markovian decision making.
abstract: Humans can learn under a wide variety of feedback conditions. Reinforcement learning (RL), where a series of rewarded decisions must be made, is a particularly important type of learning. Computational and behavioral studies of RL have focused mainly on Markovian decision processes, where the next state depends on only the current state and action. Little is known about non-Markovian decision making, where the next state depends on more than the current state and action. Learning is non-Markovian, for example, when there is no unique mapping between actions and feedback. We have produced a model based on spiking neurons that can handle these non-Markovian conditions by performing policy gradient descent [1]. Here, we examine the model's performance and compare it with human learning and a Bayes optimal reference, which provides an upper-bound on performance. We find that in all cases, our population of spiking neurons model well-describes human performance.
authors.affiliation: Brain Mind Institute, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland.
authors.forename: Aaron Michael
authors.initials: AM
authors.lastname: Clarke
authors.affiliation: Department of Physiology, University of Berne, Berne, Switzerland; Department of Statistics, Columbia University, New York, NY, USA.
authors.forename: Johannes
authors.initials: J
authors.lastname: Friedrich
authors.affiliation: Centre National de la Recherche Scientifique, Paris, France, Université Paris Descartes, Centre de Neurophysique; Physiologie et Pathologie, Paris, France; Departments of Statistics and Neurobiology, University of Chicago, Chicago, Illinois, United States of America.
authors.forename: Elisa M
authors.initials: EM
authors.lastname: Tartaglia
authors.affiliation: Brain Mind Institute, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland.
authors.forename: Silvia
authors.initials: S
authors.lastname: Marchesotti
authors.affiliation: Department of Physiology, University of Berne, Berne, Switzerland.
authors.forename: Walter
authors.initials: W
authors.lastname: Senn
authors.affiliation: Brain Mind Institute, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland.
authors.forename: Michael H
authors.initials: MH
authors.lastname: Herzog
mesh_terms: D001499:Bayes Theorem; D003198:Computer Simulation; D003657:Decision Making; D003661:Decision Support Techniques; D000068156:Formative Feedback; D006801:Humans; D008390:Markov Chains; D008959:Models, Neurological; D016571:Neural Networks, Computer