title: Seeing is believing: neural representations of visual stimuli in human auditory cortex correlate with illusory auditory perceptions.
abstract: In interpersonal communication, the listener can often see as well as hear the speaker. Visual stimuli can subtly change a listener's auditory perception, as in the McGurk illusion, in which perception of a phoneme's auditory identity is changed by a concurrent video of a mouth articulating a different phoneme. Studies have yet to link visual influences on the neural representation of language with subjective language perception. Here we show that vision influences the electrophysiological representation of phonemes in human auditory cortex prior to the presentation of the auditory stimulus. We used the McGurk effect to dissociate the subjective perception of phonemes from the auditory stimuli. With this paradigm we demonstrate that neural representations in auditory cortex are more closely correlated with the visual stimuli of mouth articulation, which drive the illusory subjective auditory perception, than the actual auditory stimuli. Additionally, information about visual and auditory stimuli transfer in the caudal-rostral direction along the superior temporal gyrus during phoneme perception as would be expected of visual information flowing from the occipital cortex into the ventral auditory processing stream. These results show that visual stimuli influence the neural representation in auditory cortex early in sensory processing and may override the subjective auditory perceptions normally generated by auditory stimuli. These findings depict a marked influence of vision on the neural processing of audition in tertiary auditory cortex and suggest a mechanistic underpinning for the McGurk effect.
authors.affiliation: Interdepartmental Program in Neuroscience, University of Utah, Salt Lake City, Utah, United States of America ; Department of Bioengineering, University of Utah, Salt Lake City, Utah, United States of America.
authors.forename: Elliot
authors.initials: E
authors.lastname: Smith
authors.forename: Scott
authors.initials: S
authors.lastname: Duede
authors.forename: Sara
authors.initials: S
authors.lastname: Hanrahan
authors.forename: Tyler
authors.initials: T
authors.lastname: Davis
authors.forename: Paul
authors.initials: P
authors.lastname: House
authors.forename: Bradley
authors.initials: B
authors.lastname: Greger
mesh_terms: D001303:Auditory Cortex / Q000502:physiology*; D001307:Auditory Perception / Q000502:physiology*; D006801:Humans; D007088:Illusions / Q000502:physiology*; D008297:Male; D010775:Photic Stimulation
references.citation: Hear Res. 2009 Dec;258(1-2):143-51
references.pmid: 19573583
references.citation: Trends Cogn Sci. 2006 Jun;10(6):278-85
references.pmid: 16713325
references.citation: J Neurosci Methods. 2010 Sep 30;192(1):146-51
references.pmid: 20637804
references.citation: Cereb Cortex. 2007 Sep;17 Suppl 1:i61-9
references.pmid: 17634387
references.citation: Proc Natl Acad Sci U S A. 2005 Jan 25;102(4):1181-6
references.pmid: 15647358
references.citation: J Neurosci. 2009 Oct 28;29(43):13613-20
references.pmid: 19864573
references.citation: Nat Neurosci. 2010 Nov;13(11):1428-32
references.pmid: 20890293
references.citation: Neurosci Lett. 1991 Jun 10;127(1):141-5
references.pmid: 1881611
references.citation: Annu Int Conf IEEE Eng Med Biol Soc. 2012;2012:5178-81
references.pmid: 23367095
references.citation: Exp Brain Res. 2007 Feb;177(2):184-95
references.pmid: 17051387
references.citation: J Neural Eng. 2010 Oct;7(5):056007
references.pmid: 20811093
references.citation: Annu Int Conf IEEE Eng Med Biol Soc. 2008;2008:4964-7
references.pmid: 19163831
references.citation: Neuroimage. 2010 Aug 15;52(2):497-507
references.pmid: 20452438
references.citation: J Neurosci. 2008 Dec 24;28(52):14301-10
references.pmid: 19109511
references.citation: J Exp Psychol Hum Percept Perform. 2011 Aug;37(4):1193-209
references.pmid: 21574741
references.citation: Nat Neurosci. 2009 Jun;12(6):718-24
references.pmid: 19471271
references.citation: Nature. 2012 May 10;485(7397):233-6
references.pmid: 22522927
references.citation: Hear Res. 2011 Jan;271(1-2):133-46
references.pmid: 20116421
references.citation: PLoS Comput Biol. 2009 Dec;5(12):e1000609
references.pmid: 20019800
references.citation: Nature. 1976 Dec 23-30;264(5588):746-8
references.pmid: 1012311
references.citation: Curr Opin Neurobiol. 1994 Apr;4(2):157-65
references.pmid: 8038571
references.citation: J Neurosci. 2005 May 18;25(20):5004-12
references.pmid: 15901781
references.citation: PLoS Biol. 2012 Jan;10(1):e1001251
references.pmid: 22303281
references.citation: Nat Rev Neurosci. 2012 May 18;13(6):407-20
references.pmid: 22595786
references.citation: J Neurosci Methods. 2010 Jan 15;185(2):293-8
references.pmid: 19836416
references.citation: Percept Psychophys. 1997 Apr;59(3):347-57
references.pmid: 9136265
references.citation: J Neural Eng. 2013 Feb;10(1):016010
references.pmid: 23283406
references.citation: J Neurosci. 2008 Apr 23;28(17):4457-69
references.pmid: 18434524