title: Credit assignment in movement-dependent reinforcement learning.
abstract: When a person fails to obtain an expected reward from an object in the environment, they face a credit assignment problem: Did the absence of reward reflect an extrinsic property of the environment or an intrinsic error in motor execution? To explore this problem, we modified a popular decision-making task used in studies of reinforcement learning, the two-armed bandit task. We compared a version in which choices were indicated by key presses, the standard response in such tasks, to a version in which the choices were indicated by reaching movements, which affords execution failures. In the key press condition, participants exhibited a strong risk aversion bias; strikingly, this bias reversed in the reaching condition. This result can be explained by a reinforcement model wherein movement errors influence decision-making, either by gating reward prediction errors or by modifying an implicit representation of motor competence. Two further experiments support the gating hypothesis. First, we used a condition in which we provided visual cues indicative of movement errors but informed the participants that trial outcomes were independent of their actual movements. The main result was replicated, indicating that the gating process is independent of participants' explicit sense of control. Second, individuals with cerebellar degeneration failed to modulate their behavior between the key press and reach conditions, providing converging evidence of an implicit influence of movement error signals on reinforcement learning. These results provide a mechanistically tractable solution to the credit assignment problem.
authors.affiliation: Department of Psychology, Princeton University, Princeton, NJ 08544; Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544; mcdougle@princeton.edu.
authors.forename: Samuel D
authors.identifier: 0000-0001-8100-4238
authors.initials: SD
authors.lastname: McDougle
authors.affiliation: Department of Psychology, University of California, Berkeley, CA 94720;
authors.forename: Matthew J
authors.initials: MJ
authors.lastname: Boggess
authors.affiliation: Department of Psychology, University of California, Berkeley, CA 94720;
authors.forename: Matthew J
authors.initials: MJ
authors.lastname: Crossley
authors.affiliation: Department of Psychology, University of California, Berkeley, CA 94720;
authors.forename: Darius
authors.initials: D
authors.lastname: Parvin
authors.affiliation: Department of Psychology, University of California, Berkeley, CA 94720; Helen Wills Neuroscience Institute, University of California, Berkeley, CA 94720.
authors.forename: Richard B
authors.initials: RB
authors.lastname: Ivry
authors.affiliation: Department of Psychology, Princeton University, Princeton, NJ 08544; Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08544;
authors.forename: Jordan A
authors.initials: JA
authors.lastname: Taylor
mesh_terms: D000293:Adolescent; D000328:Adult; D003657:Decision Making / Q000502:physiology*; D006801:Humans; D007858:Learning / Q000502:physiology*; D008297:Male; D008954:Models, Biological; D012201:Reward
keywords: cerebellum; decision-making; reinforcement learning; reward prediction error; sensory prediction error