Neural correlates of bimodal speech and gesture comprehension.
The present study examined the neural correlates of speech and hand gesture comprehension in a naturalistic context. Fifteen participants watched audiovisual segments of speech and gesture while event-related potentials (ERPs) were recorded to the speech. Gesture influenced the ERPs to the speech. Specifically, there was a right-lateralized N400 effect-reflecting semantic integration-when gestures mismatched versus matched the speech. In addition, early sensory components in bilateral occipital and frontal sites differentiated speech accompanied by matching versus non-matching gestures. These results suggest that hand gestures may be integrated with speech at early and late stages of language processing.
Kelly
Spencer D
SD
Psychology Department, Colgate University, Hamilton, NY 13346-1338, USA. skelly@mail.colgate.edu
Kravitz
Corinne
C
Hopkins
Michael
M
D000328:Adult; D001288:Attention / Q000502:physiology*; D002540:Cerebral Cortex / Q000502:physiology; D032882:Comprehension / Q000502:physiology*; D004569:Electroencephalography; D005071:Evoked Potentials / Q000502:physiology; D005260:Female; D005868:Gestures; D006801:Humans; D008297:Male; D011578:Psycholinguistics; D012660:Semantics; D013067:Speech Perception / Q000502:physiology*; D014796:Visual Perception / Q000502:physiology*