title: Computing reward-prediction error: an integrated account of cortical timing and basal-ganglia pathways for appetitive and aversive learning.
abstract: There are two prevailing notions regarding the involvement of the corticobasal ganglia system in value-based learning: (i) the direct and indirect pathways of the basal ganglia are crucial for appetitive and aversive learning, respectively, and (ii) the activity of midbrain dopamine neurons represents reward-prediction error. Although (ii) constitutes a critical assumption of (i), it remains elusive how (ii) holds given (i), with the basal-ganglia influence on the dopamine neurons. Here we present a computational neural-circuit model that potentially resolves this issue. Based on the latest analyses of the heterogeneous corticostriatal neurons and connections, our model posits that the direct and indirect pathways, respectively, represent the values of upcoming and previous actions, and up-regulate and down-regulate the dopamine neurons via the basal-ganglia output nuclei. This explains how the difference between the upcoming and previous values, which constitutes the core of reward-prediction error, is calculated. Simultaneously, it predicts that blockade of the direct/indirect pathway causes a negative/positive shift of reward-prediction error and thereby impairs learning from positive/negative error, i.e. appetitive/aversive learning. Through simulation of reward-reversal learning and punishment-avoidance learning, we show that our model could indeed account for the experimentally observed features that are suggested to support notion (i) and could also provide predictions on neural activity. We also present a behavioral prediction of our model, through simulation of inter-temporal choice, on how the balance between the two pathways relates to the subject's time preference. These results indicate that our model, incorporating the heterogeneity of the cortical influence on the basal ganglia, is expected to provide a closed-circuit mechanistic understanding of appetitive/aversive learning.
authors.affiliation: Physical and Health Education, Graduate School of Education, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-0033, Japan.
authors.forename: Kenji
authors.initials: K
authors.lastname: Morita
authors.affiliation: Division of Cerebral Circuitry, National Institute for Physiological Sciences, Okazaki, Japan.|Department of Physiological Sciences, SOKENDAI (The Graduate University for Advanced Studies), Okazaki, Japan.|Japan Science and Technology Agency, Core Research for Evolutional Science and Technology, Tokyo, Japan.
authors.forename: Yasuo
authors.initials: Y
authors.lastname: Kawaguchi
mesh_terms: D000818:Animals; D001070:Appetitive Behavior / Q000502:physiology*; D001362:Avoidance Learning / Q000502:physiology*; D001479:Basal Ganglia / Q000166:cytology* / Q000502:physiology; D002540:Cerebral Cortex / Q000166:cytology* / Q000502:physiology; D002755:Choice Behavior; D003198:Computer Simulation; D006801:Humans; D008959:Models, Neurological; D009415:Nerve Net / Q000502:physiology; D009434:Neural Pathways / Q000502:physiology; D009473:Neuronal Plasticity / Q000187:drug effects / Q000502:physiology; D009474:Neurons / Q000502:physiology*; D011237:Predictive Value of Tests; D011336:Probability; D012201:Reward; D013997:Time Factors
keywords: corticostriatal; direct pathway; dopamine; indirect pathway; reinforcement learning