title: The grammar of visual narrative: Neural evidence for constituent structure in sequential image comprehension.
abstract: Constituent structure has long been established as a central feature of human language. Analogous to how syntax organizes words in sentences, a narrative grammar organizes sequential images into hierarchic constituents. Here we show that the brain draws upon this constituent structure to comprehend wordless visual narratives. We recorded neural responses as participants viewed sequences of visual images (comics strips) in which blank images either disrupted individual narrative constituents or fell at natural constituent boundaries. A disruption of either the first or the second narrative constituent produced a left-lateralized anterior negativity effect between 500 and 700ms. Disruption of the second constituent also elicited a posteriorly-distributed positivity (P600) effect. These neural responses are similar to those associated with structural violations in language and music. These findings provide evidence that comprehenders use a narrative structure to comprehend visual sequences and that the brain engages similar neurocognitive mechanisms to build structure across multiple domains.
authors.affiliation: Department of Psychology, Tufts University, 490 Boston Ave, Medford, MA 02155, USA; Department of Cognitive Science, University of California, San Diego, 9500 Gilman Dr. Dept 0526, La Jolla, CA 92093-0526, USA. Electronic address: neilcohn@visuallanguagelab.com.
authors.forename: Neil
authors.initials: N
authors.lastname: Cohn
authors.affiliation: Department of Philosophy and Center for Cognitive Studies, Tufts University, 115 Miner Hall, Medford, MA 02155, USA; Department of Psychology, Tufts University, 490 Boston Ave, Medford, MA 02155, USA.
authors.forename: Ray
authors.initials: R
authors.lastname: Jackendoff
authors.affiliation: Department of Psychology, Tufts University, 490 Boston Ave, Medford, MA 02155, USA.
authors.forename: Phillip J
authors.initials: PJ
authors.lastname: Holcomb
authors.affiliation: Department of Psychology, Tufts University, 490 Boston Ave, Medford, MA 02155, USA; Department of Psychiatry and Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Bldg 149, 13th Street, Charlestown, MA 02129, USA.
authors.forename: Gina R
authors.initials: GR
authors.lastname: Kuperberg
mesh_terms: D001921:Brain / Q000502:physiology*; D032882:Comprehension / Q000502:physiology*; D004569:Electroencephalography; D005071:Evoked Potentials / Q000502:physiology*; D005260:Female; D006801:Humans; D007802:Language; D008297:Male; D033262:Narration; D010775:Photic Stimulation; D014796:Visual Perception / Q000502:physiology*; D055815:Young Adult
keywords: Comics; Constituent structure; ERPs; Grammar; Narrative; Visual language