title: Use of frontal lobe hemodynamics as reinforcement signals to an adaptive controller.
abstract: Decision-making ability in the frontal lobe (among other brain structures) relies on the assignment of value to states of the animal and its environment. Then higher valued states can be pursued and lower (or negative) valued states avoided. The same principle forms the basis for computational reinforcement learning controllers, which have been fruitfully applied both as models of value estimation in the brain, and as artificial controllers in their own right. This work shows how state desirability signals decoded from frontal lobe hemodynamics, as measured with near-infrared spectroscopy (NIRS), can be applied as reinforcers to an adaptable artificial learning agent in order to guide its acquisition of skills. A set of experiments carried out on an alert macaque demonstrate that both oxy- and deoxyhemoglobin concentrations in the frontal lobe show differences in response to both primarily and secondarily desirable (versus undesirable) stimuli. This difference allows a NIRS signal classifier to serve successfully as a reinforcer for an adaptive controller performing a virtual tool-retrieval task. The agent's adaptability allows its performance to exceed the limits of the NIRS classifier decoding accuracy. We also show that decoding state desirabilities is more accurate when using relative concentrations of both oxyhemoglobin and deoxyhemoglobin, rather than either species alone.
authors.affiliation: Biomedical Engineering Program, SUNY Downstate Medical Center and NYU Polytechnic, Brooklyn, New York, USA. marcello.distasio@downstate.edu
authors.forename: Marcello M
authors.initials: MM
authors.lastname: DiStasio
authors.forename: Joseph T
authors.initials: JT
authors.lastname: Francis
mesh_terms: D000222:Adaptation, Physiological / Q000502:physiology*; D000465:Algorithms; D000818:Animals; D005625:Frontal Lobe / Q000502:physiology*; D006439:Hemodynamics; D006454:Hemoglobins / Q000378:metabolism; D007858:Learning / Q000502:physiology; D008253:Macaca mulatta; D008297:Male; D012201:Reward; D012815:Signal Processing, Computer-Assisted; D013055:Spectrophotometry, Infrared
chemical_list: D006454:Hemoglobins
references.citation: Brain Res. 1971 Aug 7;31(1):35-46
references.pmid: 4998921
references.citation: Science. 2004 Apr 9;304(5668):307-10
references.pmid: 15073380
references.citation: Neuroimage. 2011 Apr 15;55(4):1435-42
references.pmid: 21256229
references.citation: J Neuroeng Rehabil. 2011 Jun 18;8:34
references.pmid: 21682906
references.citation: Nat Neurosci. 1998 Aug;1(4):286-9
references.pmid: 10195161
references.citation: J Neurophysiol. 1989 Feb;61(2):331-49
references.pmid: 2918358
references.citation: Philos Trans R Soc Lond B Biol Sci. 1998 Nov 29;353(1377):1819-28
references.pmid: 9854254
references.citation: Nat Neurosci. 1999 Feb;2(2):176-85
references.pmid: 10195203
references.citation: J Neurosci. 2011 Dec 7;31(49):17772-87
references.pmid: 22159094
references.citation: Trends Neurosci. 1991 Jan;14(1):21-7
references.pmid: 1709528
references.citation: J Neurosci. 2008 Nov 5;28(45):11703-11
references.pmid: 18987206
references.citation: J Biomed Opt. 2011 Aug;16(8):087005
references.pmid: 21895332
references.citation: Science. 2004 Oct 15;306(5695):503-7
references.pmid: 15486304
references.citation: Neurosci Lett. 2007 Apr 6;416(1):55-60
references.pmid: 17316990
references.citation: Neurosci Res. 1984 Apr;1(2):89-103
references.pmid: 6536892
references.citation: Nature. 2002 Apr 11;416(6881):632-6
references.pmid: 11948351
references.citation: PLoS Comput Biol. 2008 Apr 04;4(4):e1000050
references.pmid: 18389057
references.citation: Magn Reson Imaging. 2006 May;24(4):495-505
references.pmid: 16677956
references.citation: Eur J Neurosci. 2003 Oct;18(7):2069-81
references.pmid: 14622240
references.citation: J Comp Neurol. 1985 Feb 22;232(4):443-55
references.pmid: 3980763
references.citation: Science. 2007 Oct 26;318(5850):594-8
references.pmid: 17962551
references.citation: Annu Int Conf IEEE Eng Med Biol Soc. 2009;2009:594-7
references.pmid: 19964231
references.citation: J Neurosci. 1996 Aug 15;16(16):5154-67
references.pmid: 8756444
references.citation: Biomed Eng Online. 2010 Mar 09;9:16
references.pmid: 20214809
references.citation: Magn Reson Med. 1998 Jun;39(6):855-64
references.pmid: 9621908
references.citation: Proc Soc Exp Biol Med. 1946 Mar;61:291-6
references.pmid: 21024171
references.citation: Cereb Cortex. 1993 May-Jun;3(3):199-222
references.pmid: 8100725
references.citation: Physiol Meas. 2004 Aug;25(4):815-22
references.pmid: 15382823
references.citation: Nature. 1993 Jun 17;363(6430):623-5
references.pmid: 8510752
references.citation: Insight J. 2006;2006:209
references.pmid: 25364771
references.citation: Br J Pharmacol. 1985 Jun;85(2):403-10
references.pmid: 3896363
references.citation: Behav Neurosci. 1989 Aug;103(4):770-8
references.pmid: 2765181
references.citation: J Neurosci Methods. 2010 May 15;188(2):187-94
references.pmid: 20153370
references.citation: J Neural Eng. 2009 Feb;6(1):016003
references.pmid: 19104138
references.citation: J Magn Reson Imaging. 2006 Mar;23(3):408-12
references.pmid: 16456823
references.citation: Science. 2003 Jul 11;301(5630):229-32
references.pmid: 12855813
references.citation: Neuron. 1999 Oct;24(2):415-25
references.pmid: 10571234
references.citation: Nature. 1996 Aug 15;382(6592):629-32
references.pmid: 8757133
references.citation: J Comp Neurol. 1985 Jun 15;236(3):315-30
references.pmid: 4056098
references.citation: Annu Rev Psychol. 2006;57:87-115
references.pmid: 16318590
references.citation: Phys Med Biol. 2003 Aug 7;48(15):2405-18
references.pmid: 12953906
references.citation: Opt Express. 2001 Sep 10;9(6):272-86
references.pmid: 19421298
references.citation: Neuroimage. 2006 Apr 15;30(3):700-12
references.pmid: 16459104
references.citation: Neurosci Biobehav Rev. 1989 Summer-Fall;13(2-3):163-70
references.pmid: 2682403
references.citation: J Neurosci. 1988 Nov;8(11):4049-68
references.pmid: 2846794
references.citation: Cardiovasc Res. 2006 Apr 1;70(1):12-21
references.pmid: 16360130
references.citation: J Neurophysiol. 2007 Feb;97(2):1621-32
references.pmid: 17122317
references.citation: Proc Natl Acad Sci U S A. 1990 Dec;87(24):9868-72
references.pmid: 2124706
references.citation: Proc Natl Acad Sci U S A. 1936 Apr;22(4):210-6
references.pmid: 16577697
references.citation: Science. 1993 Jun 25;260(5116):1955-8
references.pmid: 8316836
references.citation: Biochim Biophys Acta. 1988 Mar 30;933(1):184-92
references.pmid: 2831976
references.citation: J Neurosci. 1990 Nov;10(11):3479-93
references.pmid: 2230939
references.citation: Nature. 2001 Jul 12;412(6843):150-7
references.pmid: 11449264
references.citation: Proc Natl Acad Sci U S A. 1996 Oct 15;93(21):12040-5
references.pmid: 8876259
references.citation: PLoS One. 2008 Mar 12;3(3):e1760
references.pmid: 18335036
references.citation: Annu Int Conf IEEE Eng Med Biol Soc. 2011;2011:5662-5
references.pmid: 22255624
references.citation: Proc Natl Acad Sci U S A. 2011 Jan 4;108(1):55-60
references.pmid: 21173266
references.citation: J Comp Neurol. 1989 Sep 22;287(4):422-45
references.pmid: 2477406
references.citation: Science. 1997 Mar 14;275(5306):1593-9
references.pmid: 9054347
references.citation: Annu Int Conf IEEE Eng Med Biol Soc. 2012;2012:3284-7
references.pmid: 23366627
references.citation: J Neurophysiol. 1998 Jan;79(1):322-33
references.pmid: 9425201
references.citation: J Cereb Blood Flow Metab. 1996 Sep;16(5):817-26
references.pmid: 8784226
references.citation: IEEE Trans Biomed Eng. 2005 May;52(5):934-8
references.pmid: 15887543
references.citation: Nature. 1996 Feb 1;379(6564):449-51
references.pmid: 8559249
references.citation: Neuron. 2009 Jul 30;63(2):244-53
references.pmid: 19640482
references.citation: J Neurophysiol. 2002 Mar;87(3):1488-98
references.pmid: 11877521