title: When a hit sounds like a kiss: An electrophysiological exploration of semantic processing in visual narrative.
abstract: Researchers have long questioned whether information presented through different sensory modalities involves distinct or shared semantic systems. We investigated uni-sensory cross-modal processing by recording event-related brain potentials to words replacing the climactic event in a visual narrative sequence (comics). We compared Onomatopoeic words, which phonetically imitate action sounds (Pow!), with Descriptive words, which describe an action (Punch!), that were (in)congruent within their sequence contexts. Across two experiments, larger N400s appeared to Anomalous Onomatopoeic or Descriptive critical panels than to their congruent counterparts, reflecting a difficulty in semantic access/retrieval. Also, Descriptive words evinced a greater late frontal positivity compared to Onomatopoetic words, suggesting that, though plausible, they may be less predictable/expected in visual narratives. Our results indicate that uni-sensory cross-model integration of word/letter-symbol strings within visual narratives elicit ERP patterns typically observed for written sentence processing, thereby suggesting the engagement of similar domain-independent integration/interpretation mechanisms.
authors.affiliation: Department of Psychology, University of Milano-Bicocca, Milan, Italy; Social and Cognitive Neuroscience Laboratory, Center for Biological Science and Health, Mackenzie Presbyterian University, São Paulo, Brazil. Electronic address: mirella.manfredi@gmail.com.
authors.forename: Mirella
authors.initials: M
authors.lastname: Manfredi
authors.affiliation: Department of Cognitive Science, University of California, San Diego, La Jolla, CA, USA; Tilburg Center for Cognition and Communication, Tilburg University, Tilburg, Netherlands.
authors.forename: Neil
authors.initials: N
authors.lastname: Cohn
authors.affiliation: Department of Cognitive Science, University of California, San Diego, La Jolla, CA, USA.
authors.forename: Marta
authors.initials: M
authors.lastname: Kutas
mesh_terms: D000293:Adolescent; D000328:Adult; D001921:Brain / Q000502:physiology*; D004569:Electroencephalography; D005071:Evoked Potentials / Q000502:physiology*; D005260:Female; D006801:Humans; D008297:Male; D033262:Narration; D010700:Phonetics; D010775:Photic Stimulation; D012660:Semantics; D013016:Sound; D055815:Young Adult
keywords: Comics; Event-related potentials; Late positivity; N400; Onomatopoeic words; Visual language; Visual narrative