title: A neural signature of hierarchical reinforcement learning.
abstract: Human behavior displays hierarchical structure: simple actions cohere into subtask sequences, which work together to accomplish overall task goals. Although the neural substrates of such hierarchy have been the target of increasing research, they remain poorly understood. We propose that the computations supporting hierarchical behavior may relate to those in hierarchical reinforcement learning (HRL), a machine-learning framework that extends reinforcement-learning mechanisms into hierarchical domains. To test this, we leveraged a distinctive prediction arising from HRL. In ordinary reinforcement learning, reward prediction errors are computed when there is an unanticipated change in the prospects for accomplishing overall task goals. HRL entails that prediction errors should also occur in relation to task subgoals. In three neuroimaging studies we observed neural responses consistent with such subgoal-related reward prediction errors, within structures previously implicated in reinforcement learning. The results reported support the relevance of HRL to the neural processes underlying hierarchical behavior.
authors.affiliation: Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08540, USA.
authors.forename: José J F
authors.initials: JJ
authors.lastname: Ribas-Fernandes
authors.forename: Alec
authors.initials: A
authors.lastname: Solway
authors.forename: Carlos
authors.initials: C
authors.lastname: Diuk
authors.forename: Joseph T
authors.initials: JT
authors.lastname: McGuire
authors.forename: Andrew G
authors.initials: AG
authors.lastname: Barto
authors.forename: Yael
authors.initials: Y
authors.lastname: Niv
authors.forename: Matthew M
authors.initials: MM
authors.lastname: Botvinick
mesh_terms: D000293:Adolescent; D000328:Adult; D001921:Brain / Q000098:blood supply / Q000502:physiology*; D001931:Brain Mapping; D058256:Brain Waves / Q000502:physiology*; D003198:Computer Simulation; D004569:Electroencephalography / Q000379:methods; D005260:Female; D006801:Humans; D007091:Image Processing, Computer-Assisted; D016014:Linear Models; D008279:Magnetic Resonance Imaging / Q000379:methods; D008297:Male; D008954:Models, Biological; D010100:Oxygen / Q000097:blood; D011597:Psychomotor Performance / Q000502:physiology; D012054:Reinforcement, Psychology; D055815:Young Adult
chemical_list: D010100:Oxygen