title: Different mechanisms for role relations versus verb-action congruence effects: evidence from ERPs in picture-sentence verification.
abstract: Extant accounts of visually situated language processing do make general predictions about visual context effects on incremental sentence comprehension; these, however, are not sufficiently detailed to accommodate potentially different visual context effects (such as a scene-sentence mismatch based on actions versus thematic role relations, e.g., (Altmann & Kamide, 2007; Knoeferle & Crocker, 2007; Taylor & Zwaan, 2008; Zwaan & Radvansky, 1998)). To provide additional data for theory testing and development, we collected event-related brain potentials (ERPs) as participants read a subject-verb-object sentence (500 ms SOA in Experiment 1 and 300 ms SOA in Experiment 2), and post-sentence verification times indicating whether or not the verb and/or the thematic role relations matched a preceding picture (depicting two participants engaged in an action). Though incrementally processed, these two types of mismatch yielded different ERP effects. Role-relation mismatch effects emerged at the subject noun as anterior negativities to the mismatching noun, preceding action mismatch effects manifest as centro-parietal N400s greater to the mismatching verb, regardless of SOAs. These two types of mismatch manipulations also yielded different effects post-verbally, correlated differently with a participant's mean accuracy, verbal working memory and visual-spatial scores, and differed in their interactions with SOA. Taken together these results clearly implicate more than a single mismatch mechanism for extant accounts of picture-sentence processing to accommodate.
authors.affiliation: Cognitive Interaction Technology Excellence Cluster, Bielefeld University, Germany. Electronic address: knoeferl@cit-ec.uni-bielefeld.de.
authors.forename: Pia
authors.initials: P
authors.lastname: Knoeferle
authors.affiliation: Department of Cognitive Science, University of California, San Diego, United States.
authors.forename: Thomas P
authors.initials: TP
authors.lastname: Urbach
authors.affiliation: Department of Cognitive Science, University of California, San Diego, United States.
authors.forename: Marta
authors.initials: M
authors.lastname: Kutas
mesh_terms: D000293:Adolescent; D000328:Adult; D001921:Brain / Q000502:physiology*; D001931:Brain Mapping; D032882:Comprehension / Q000502:physiology*; D004569:Electroencephalography; D005071:Evoked Potentials / Q000502:physiology*; D005260:Female; D006801:Humans; D007802:Language; D008297:Male; D008570:Memory, Short-Term / Q000502:physiology; D010775:Photic Stimulation; D011932:Reading; D014796:Visual Perception / Q000502:physiology; D055815:Young Adult
keywords: Event-related brain potentials; Sentence–picture verification; Situated language processing accounts; Visual context effects