title: Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning.
abstract: Current learning theory provides a comprehensive description of how humans and other animals learn, and places behavioral flexibility and automaticity at heart of adaptive behaviors. However, the computations supporting the interactions between goal-directed and habitual decision-making systems are still poorly understood. Previous functional magnetic resonance imaging (fMRI) results suggest that the brain hosts complementary computations that may differentially support goal-directed and habitual processes in the form of a dynamical interplay rather than a serial recruitment of strategies. To better elucidate the computations underlying flexible behavior, we develop a dual-system computational model that can predict both performance (i.e., participants' choices) and modulations in reaction times during learning of a stimulus-response association task. The habitual system is modeled with a simple Q-Learning algorithm (QL). For the goal-directed system, we propose a new Bayesian Working Memory (BWM) model that searches for information in the history of previous trials in order to minimize Shannon entropy. We propose a model for QL and BWM coordination such that the expensive memory manipulation is under control of, among others, the level of convergence of the habitual learning. We test the ability of QL or BWM alone to explain human behavior, and compare them with the performance of model combinations, to highlight the need for such combinations to explain behavior. Two of the tested combination models are derived from the literature, and the latter being our new proposal. In conclusion, all subjects were better explained by model combinations, and the majority of them are explained by our new coordination proposal.
authors.affiliation: Sorbonne Université, Université Pierre et Marie Curie, Univ Paris 06, UMR 7222, Institut des Systèmes Intelligents et de Robotique Paris, France ; Centre National de la Recherche Scientifique, UMR 7222, ISIR Paris, France.
authors.forename: Guillaume
authors.initials: G
authors.lastname: Viejo
authors.affiliation: Sorbonne Université, Université Pierre et Marie Curie, Univ Paris 06, UMR 7222, Institut des Systèmes Intelligents et de Robotique Paris, France ; Centre National de la Recherche Scientifique, UMR 7222, ISIR Paris, France.
authors.forename: Mehdi
authors.initials: M
authors.lastname: Khamassi
authors.affiliation: Institut de Neurosciences de la Timone, UMR 7289, Centre National de la Recherche Scientifique - Aix Marseille Université Marseille, France.
authors.forename: Andrea
authors.initials: A
authors.lastname: Brovelli
authors.affiliation: Sorbonne Université, Université Pierre et Marie Curie, Univ Paris 06, UMR 7222, Institut des Systèmes Intelligents et de Robotique Paris, France ; Centre National de la Recherche Scientifique, UMR 7222, ISIR Paris, France.
authors.forename: Benoît
authors.initials: B
authors.lastname: Girard
keywords: action selection; behavior; decision-making; multi-objective optimization; reaction times; reinforcement learning; working-memory