title: Goal-Directed and Habit-Like Modulations of Stimulus Processing during Reinforcement Learning.
abstract: Recent research has shown that perceptual processing of stimuli previously associated with high-value rewards is automatically prioritized even when rewards are no longer available. It has been hypothesized that such reward-related modulation of stimulus salience is conceptually similar to an "attentional habit." Recording event-related potentials in humans during a reinforcement learning task, we show strong evidence in favor of this hypothesis. Resistance to outcome devaluation (the defining feature of a habit) was shown by the stimulus-locked P1 component, reflecting activity in the extrastriate visual cortex. Analysis at longer latencies revealed a positive component (corresponding to the P3b, from 550-700 ms) sensitive to outcome devaluation. Therefore, distinct spatiotemporal patterns of brain activity were observed corresponding to habitual and goal-directed processes. These results demonstrate that reinforcement learning engages both attentional habits and goal-directed processes in parallel. Consequences for brain and computational models of reinforcement learning are discussed.SIGNIFICANCE STATEMENTThe human attentional network adapts to detect stimuli that predict important rewards. A recent hypothesis suggests that the visual cortex automatically prioritizes reward-related stimuli, driven by cached representations of reward value; that is, stimulus-response habits. Alternatively, the neural system may track the current value of the predicted outcome. Our results demonstrate for the first time that visual cortex activity is increased for reward-related stimuli even when the rewarding event is temporarily devalued. In contrast, longer-latency brain activity was specifically sensitive to transient changes in reward value. Therefore, we show that both habit-like attention and goal-directed processes occur in the same learning episode at different latencies. This result has important consequences for computational models of reinforcement learning.
authors.affiliation: School of Psychology, UNSW Australia, Sydney, New South Wales 2052, Australia d.luque@unsw.edu.au.
authors.forename: David
authors.identifier: 0000-0002-3457-9204
authors.initials: D
authors.lastname: Luque
authors.affiliation: School of Psychology, UNSW Australia, Sydney, New South Wales 2052, Australia.
authors.forename: Tom
authors.initials: T
authors.lastname: Beesley
authors.affiliation: School of Psychology, UNSW Australia, Sydney, New South Wales 2052, Australia.
authors.forename: Richard W
authors.identifier: 0000-0002-5018-1239
authors.initials: RW
authors.lastname: Morris
authors.affiliation: School of Psychology, UNSW Australia, Sydney, New South Wales 2052, Australia.
authors.forename: Bradley N
authors.initials: BN
authors.lastname: Jack
authors.affiliation: School of Psychology, UNSW Australia, Sydney, New South Wales 2052, Australia.
authors.forename: Oren
authors.identifier: 0000-0002-9833-9998
authors.initials: O
authors.lastname: Griffiths
authors.affiliation: School of Psychology, UNSW Australia, Sydney, New South Wales 2052, Australia.
authors.forename: Thomas J
authors.initials: TJ
authors.lastname: Whitford
authors.affiliation: School of Psychology, UNSW Australia, Sydney, New South Wales 2052, Australia.
authors.forename: Mike E
authors.initials: ME
authors.lastname: Le Pelley
mesh_terms: D000328:Adult; D001288:Attention / Q000502:physiology*; D002755:Choice Behavior / Q000502:physiology*; D003463:Cues; D005260:Female; D006040:Goals; D006184:Habits; D006801:Humans; D008297:Male; D009415:Nerve Net / Q000502:physiology; D011930:Reaction Time / Q000502:physiology; D012054:Reinforcement, Psychology; D014793:Visual Cortex / Q000502:physiology*
keywords: attention; event-related potentials; goal-directed; habit; learning; reward