title: Distributed neural signatures of natural audiovisual speech and music in the human auditory cortex.
abstract: During a conversation or when listening to music, auditory and visual information are combined automatically into audiovisual objects. However, it is still poorly understood how specific type of visual information shapes neural processing of sounds in lifelike stimulus environments. Here we applied multi-voxel pattern analysis to investigate how naturally matching visual input modulates supratemporal cortex activity during processing of naturalistic acoustic speech, singing and instrumental music. Bayesian logistic regression classifiers with sparsity-promoting priors were trained to predict whether the stimulus was audiovisual or auditory, and whether it contained piano playing, speech, or singing. The predictive performances of the classifiers were tested by leaving one participant at a time for testing and training the model using the remaining 15 participants. The signature patterns associated with unimodal auditory stimuli encompassed distributed locations mostly in the middle and superior temporal gyrus (STG/MTG). A pattern regression analysis, based on a continuous acoustic model, revealed that activity in some of these MTG and STG areas were associated with acoustic features present in speech and music stimuli. Concurrent visual stimulus modulated activity in bilateral MTG (speech), lateral aspect of right anterior STG (singing), and bilateral parietal opercular cortex (piano). Our results suggest that specific supratemporal brain areas are involved in processing complex natural speech, singing, and piano playing, and other brain areas located in anterior (facial speech) and posterior (music-related hand actions) supratemporal cortex are influenced by related visual information. Those anterior and posterior supratemporal areas have been linked to stimulus identification and sensory-motor integration, respectively.
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland; Advanced Magnetic Imaging (AMI) Centre, School of Science, Aalto University, Finland; Institute of Behavioural Sciences, Division of Cognitive and Neuropsychology, University of Helsinki, Finland.
authors.forename: Juha
authors.initials: J
authors.lastname: Salmi
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland.
authors.forename: Olli-Pekka
authors.initials: OP
authors.lastname: Koistinen
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland.
authors.forename: Enrico
authors.initials: E
authors.lastname: Glerean
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland.
authors.forename: Pasi
authors.initials: P
authors.lastname: Jylänki
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland.
authors.forename: Aki
authors.initials: A
authors.lastname: Vehtari
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland.
authors.forename: Iiro P
authors.initials: IP
authors.lastname: Jääskeläinen
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland.
authors.forename: Sasu
authors.initials: S
authors.lastname: Mäkelä
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland; Turku PET Centre, University of Turku, Finland.
authors.forename: Lauri
authors.initials: L
authors.lastname: Nummenmaa
authors.affiliation: DocMus Unit, Sibelius Academy, Helsinki, Finland.
authors.forename: Katarina
authors.initials: K
authors.lastname: Nummi-Kuisma
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland.
authors.forename: Ilari
authors.initials: I
authors.lastname: Nummi
authors.affiliation: Department of Neuroscience and Biomedical Engineering (NBE), School of Science, Aalto University, Finland. Electronic address: Mikko.Sams@aalto.fi.
authors.forename: Mikko
authors.initials: M
authors.lastname: Sams
mesh_terms: D000328:Adult; D001303:Auditory Cortex / Q000502:physiology*; D001307:Auditory Perception / Q000502:physiology*; D001931:Brain Mapping / Q000379:methods*; D005260:Female; D006801:Humans; D008279:Magnetic Resonance Imaging; D008297:Male; D009146:Music; D013067:Speech Perception / Q000502:physiology; D014796:Visual Perception / Q000502:physiology*; D055815:Young Adult
keywords: Audiovisual; Multi-voxel pattern analysis; Music; Speech; fMRI