title: Reinforcement learning in multidimensional environments relies on attention mechanisms.
abstract: In recent years, ideas from the computational field of reinforcement learning have revolutionized the study of learning in the brain, famously providing new, precise theories of how dopamine affects learning in the basal ganglia. However, reinforcement learning algorithms are notorious for not scaling well to multidimensional environments, as is required for real-world learning. We hypothesized that the brain naturally reduces the dimensionality of real-world problems to only those dimensions that are relevant to predicting reward, and conducted an experiment to assess by what algorithms and with what neural mechanisms this "representation learning" process is realized in humans. Our results suggest that a bilateral attentional control network comprising the intraparietal sulcus, precuneus, and dorsolateral prefrontal cortex is involved in selecting what dimensions are relevant to the task at hand, effectively updating the task representation through trial and error. In this way, cortical attention mechanisms interact with learning in the basal ganglia to solve the "curse of dimensionality" in reinforcement learning.
authors.affiliation: Department of Psychology and Neuroscience Institute, Princeton University, Princeton, New Jersey 08540, yael@princeton.edu.
authors.forename: Yael
authors.identifier: http://orcid.org/0000-0002-0259-8371
authors.initials: Y
authors.lastname: Niv
authors.affiliation: Department of Psychology and Neuroscience Institute, Princeton University, Princeton, New Jersey 08540.
authors.forename: Reka
authors.identifier: http://orcid.org/0000-0003-3706-7623
authors.initials: R
authors.lastname: Daniel
authors.affiliation: Department of Psychology and Neuroscience Institute, Princeton University, Princeton, New Jersey 08540.
authors.forename: Andra
authors.initials: A
authors.lastname: Geana
authors.affiliation: Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, Massachusetts 02139.
authors.forename: Samuel J
authors.initials: SJ
authors.lastname: Gershman
authors.affiliation: Department of Psychology, Stanford University, Stanford, California 94305, and.
authors.forename: Yuan Chang
authors.initials: YC
authors.lastname: Leong
authors.affiliation: Department of Psychology and Neuroscience Institute, Princeton University, Princeton, New Jersey 08540.
authors.forename: Angela
authors.initials: A
authors.lastname: Radulescu
authors.affiliation: Department of Psychology and Cognitive Science Program, University of Arizona, Tucson, Arizona 85721.
authors.forename: Robert C
authors.initials: RC
authors.lastname: Wilson
mesh_terms: D000293:Adolescent; D000328:Adult; D001288:Attention / Q000502:physiology*; D002755:Choice Behavior / Q000502:physiology*; D004777:Environment; D005260:Female; D006801:Humans; D007858:Learning / Q000502:physiology*; D008297:Male; D010775:Photic Stimulation / Q000379:methods*; D012054:Reinforcement, Psychology; D055815:Young Adult
keywords: attention; fMRI; frontoparietal network; model comparison; reinforcement learning; representation learning