title: Dopamine reward prediction errors reflect hidden-state inference across time.
abstract: Midbrain dopamine neurons signal reward prediction error (RPE), or actual minus expected reward. The temporal difference (TD) learning model has been a cornerstone in understanding how dopamine RPEs could drive associative learning. Classically, TD learning imparts value to features that serially track elapsed time relative to observable stimuli. In the real world, however, sensory stimuli provide ambiguous information about the hidden state of the environment, leading to the proposal that TD learning might instead compute a value signal based on an inferred distribution of hidden states (a 'belief state'). Here we asked whether dopaminergic signaling supports a TD learning framework that operates over hidden states. We found that dopamine signaling showed a notable difference between two tasks that differed only with respect to whether reward was delivered in a deterministic manner. Our results favor an associative learning rule that combines cached values with hidden-state inference.
authors.affiliation: Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, Cambridge, Massachusetts, USA.
authors.forename: Clara Kwon
authors.initials: CK
authors.lastname: Starkweather
authors.affiliation: Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, Cambridge, Massachusetts, USA.|Center for Brain Science, Department of Psychology, Harvard University, Cambridge, Massachusetts, USA.
authors.forename: Benedicte M
authors.initials: BM
authors.lastname: Babayan
authors.affiliation: Center for Brain Science, Department of Molecular and Cellular Biology, Harvard University, Cambridge, Massachusetts, USA.
authors.forename: Naoshige
authors.identifier: http://orcid.org/0000-0002-5755-9409
authors.initials: N
authors.lastname: Uchida
authors.affiliation: Center for Brain Science, Department of Psychology, Harvard University, Cambridge, Massachusetts, USA.
authors.forename: Samuel J
authors.identifier: http://orcid.org/0000-0002-6546-3298
authors.initials: SJ
authors.lastname: Gershman
mesh_terms: D000818:Animals; D001245:Association Learning / Q000502:physiology*; D059290:Dopaminergic Neurons / Q000502:physiology*; D008297:Male; D051379:Mice; D008959:Models, Neurological; D012201:Reward; D013997:Time Factors; D017557:Ventral Tegmental Area / Q000502:physiology