title: Reinforcement learning of two-joint virtual arm reaching in a computer model of sensorimotor cortex.
abstract: Neocortical mechanisms of learning sensorimotor control involve a complex series of interactions at multiple levels, from synaptic mechanisms to cellular dynamics to network connectomics. We developed a model of sensory and motor neocortex consisting of 704 spiking model neurons. Sensory and motor populations included excitatory cells and two types of interneurons. Neurons were interconnected with AMPA/NMDA and GABAA synapses. We trained our model using spike-timing-dependent reinforcement learning to control a two-joint virtual arm to reach to a fixed target. For each of 125 trained networks, we used 200 training sessions, each involving 15 s reaches to the target from 16 starting positions. Learning altered network dynamics, with enhancements to neuronal synchrony and behaviorally relevant information flow between neurons. After learning, networks demonstrated retention of behaviorally relevant memories by using proprioceptive information to perform reach-to-target from multiple starting positions. Networks dynamically controlled which joint rotations to use to reach a target, depending on current arm position. Learning-dependent network reorganization was evident in both sensory and motor populations: learned synaptic weights showed target-specific patterning optimized for particular reach movements. Our model embodies an integrative hypothesis of sensorimotor cortical learning that could be used to interpret future electrophysiological data recorded in vivo from sensorimotor learning experiments. We used our model to make the following predictions: learning enhances synchrony in neuronal populations and behaviorally relevant information flow across neuronal populations, enhanced sensory processing aids task-relevant motor performance and the relative ease of a particular movement in vivo depends on the amount of sensory information required to complete the movement.
authors.affiliation: Department of Neurobiology, Yale University School of Medicine, New Haven, CT 06510, U.S.A., and Department of Physiology and Pharmacology, SUNY Downstate, Brooklyn, NY 11203, U.S.A. samuel.neymotin@yale.edu.
authors.forename: Samuel A
authors.initials: SA
authors.lastname: Neymotin
authors.forename: George L
authors.initials: GL
authors.lastname: Chadderdon
authors.forename: Cliff C
authors.initials: CC
authors.lastname: Kerr
authors.forename: Joseph T
authors.initials: JT
authors.lastname: Francis
authors.forename: William W
authors.initials: WW
authors.lastname: Lytton
mesh_terms: D001132:Arm / Q000294:innervation; D001185:Artificial Intelligence; D003198:Computer Simulation; D006801:Humans; D011597:Psychomotor Performance / Q000502:physiology*; D013003:Somatosensory Cortex / Q000502:physiology*
references.citation: Exp Brain Res. 2008 Mar;185(3):359-81
references.pmid: 18251019
references.citation: Curr Opin Neurobiol. 2003 Apr;13(2):225-31
references.pmid: 12744978
references.citation: Infant Behav Dev. 2009 Jan;32(1):44-58
references.pmid: 19081142
references.citation: J Neurosci. 2011 Feb 16;31(7):2481-7
references.pmid: 21325515
references.citation: J Am Med Inform Assoc. 1996 Nov-Dec;3(6):389-98
references.pmid: 8930855
references.citation: Biol Cybern. 1986;54(1):29-40
references.pmid: 3719028
references.citation: Nat Neurosci. 2009 Jul;12(7):927-31
references.pmid: 19525945
references.citation: Front Comput Neurosci. 2013 Apr 25;7:39
references.pmid: 23630492
references.citation: PLoS One. 2010 Jun 15;5(6):e11125
references.pmid: 20559553
references.citation: PLoS Comput Biol. 2008 May 30;4(5):e1000078
references.pmid: 18516226
references.citation: J Neurosci. 2011 Aug 10;31(32):11733-43
references.pmid: 21832203
references.citation: Exp Brain Res. 1999 Aug;127(3):259-69
references.pmid: 10452213
references.citation: Cereb Cortex. 2003 Jan;13(1):5-14
references.pmid: 12466210
references.citation: Neuron. 2012 Jul 26;75(2):209-17
references.pmid: 22841307
references.citation: Neuron. 2003 Dec 18;40(6):1063-73
references.pmid: 14687542
references.citation: Neural Comput. 2004 Feb;16(2):251-75
references.pmid: 15006096
references.citation: PLoS One. 2012;7(10):e47251
references.pmid: 23094042
references.citation: J Neurosci Methods. 2008 Jun 30;171(2):309-15
references.pmid: 18452996
references.citation: PLoS One. 2011 Mar 14;6(3):e14760
references.pmid: 21423797
references.citation: Int J Dev Neurosci. 2009 Aug;27(5):415-21
references.pmid: 19446627
references.citation: PLoS Comput Biol. 2006 Oct 27;2(10):e144
references.pmid: 17069456
references.citation: Neural Comput. 2007 Jun;19(6):1468-502
references.pmid: 17444757
references.citation: Cereb Cortex. 2002 Sep;12(9):936-53
references.pmid: 12183393
references.citation: Neural Comput. 2009 Feb;21(2):301-39
references.pmid: 19196231
references.citation: J Neurosci. 2011 Jan 19;31(3):878-82
references.pmid: 21248111
references.citation: Cereb Cortex. 1995 Mar-Apr;5(2):95-110
references.pmid: 7620294
references.citation: Artif Life. 2005 Winter-Spring;11(1-2):63-77
references.pmid: 15811220
references.citation: J Neurosci. 2010 Oct 13;30(41):13760-5
references.pmid: 20943916
references.citation: Neural Netw. 2002 Jun-Jul;15(4-6):507-21
references.pmid: 12371508
references.citation: PLoS One. 2009 Sep 17;4(9):e7082
references.pmid: 19759902
references.citation: BMC Syst Biol. 2007 Jun 13;1:28
references.pmid: 17567903
references.citation: J Neurophysiol. 2007 Feb;97(2):1566-87
references.pmid: 17093119
references.citation: Front Comput Neurosci. 2011 Apr 19;5:19
references.pmid: 21541305
references.citation: PLoS Comput Biol. 2005 Sep;1(4):e42
references.pmid: 16201007
references.citation: Cereb Cortex. 1998 Jun;8(4):346-61
references.pmid: 9651130
references.citation: PLoS Biol. 2010 Jun 22;8(6):e1000403
references.pmid: 20585373
references.citation: J Neurophysiol. 2007 Dec;98(6):3648-65
references.pmid: 17928565
references.citation: J Comput Neurosci. 2011 Feb;30(1):69-84
references.pmid: 20556639
references.citation: J Neurosci. 2010 Mar 31;30(13):4613-25
references.pmid: 20357112
references.citation: Neuron. 2006 Oct 5;52(1):155-68
references.pmid: 17015233
references.citation: J Clin Neurophysiol. 2007 Apr;24(2):175-81
references.pmid: 17414973
references.citation: Neural Comput. 2008 Nov;20(11):2745-56
references.pmid: 18533821
references.citation: Neuron. 2011 Aug 11;71(3):555-64
references.pmid: 21835350
references.citation: Science. 1994 Sep 23;265(5180):1826-31
references.pmid: 8091209
references.citation: Science. 2006 Aug 25;313(5790):1141-4
references.pmid: 16931766
references.citation: PLoS Comput Biol. 2012;8(4):e1002493
references.pmid: 22536151
references.citation: Curr Opin Neurobiol. 2002 Apr;12(2):217-22
references.pmid: 12015240
references.citation: Behav Neurosci. 2006 Jun;120(3):497-517
references.pmid: 16768602
references.citation: Science. 1988 Jun 24;240(4860):1802
references.pmid: 17842436
references.citation: Science. 2004 Dec 10;306(5703):1940-3
references.pmid: 15528409
references.citation: IEEE Trans Neural Syst Rehabil Eng. 2012 Mar;20(2):153-60
references.pmid: 22180517
references.citation: Nat Neurosci. 2000 Sep;3(9):919-26
references.pmid: 10966623
references.citation: Science. 2008 Aug 8;321(5890):848-51
references.pmid: 18687967
references.citation: J Chem Neuroanat. 2010 Mar;39(2):112-23
references.pmid: 19900537
references.citation: Nat Rev Neurosci. 2008 Aug;9(8):626-37
references.pmid: 18594562
references.citation: J Neurosci. 2011 Apr 13;31(15):5855-64
references.pmid: 21490227
references.citation: J Neurophysiol. 2007 Mar;97(3):2533-43
references.pmid: 17202243
references.citation: Biol Cybern. 2002 Dec;87(5-6):392-403
references.pmid: 12461629
references.citation: Anim Behav. 2000 Nov;60(5):545-558
references.pmid: 11082225
references.citation: Neurosci Biobehav Rev. 2006;30(1):1-23
references.pmid: 15935475
references.citation: Cereb Cortex. 2007 Oct;17(10):2443-52
references.pmid: 17220510
references.citation: Neurosci Res. 2005 Oct;53(2):95-103
references.pmid: 16054257