title: Visual Presentation Effects on Identification of Multiple Environmental Sounds.
abstract: This study examined how the contents and timing of a visual stimulus affect the identification of mixed sounds recorded in a daily life environment. For experiments, we presented four environment sounds as auditory stimuli for 5 s along with a picture or a written word as a visual stimulus that might or might not denote the source of one of the four sounds. Three conditions of temporal relations between the visual stimuli and sounds were used. The visual stimulus was presented either: (a) for 5 s simultaneously with the sound; (b) for 5 s, 1 s before the sound (SOA between the audio and visual stimuli was 6 s); or (c) for 33 ms, 1 s before the sound (SOA was 1033 ms). Participants reported all identifiable sounds for those audio-visual stimuli. To characterize the effects of visual stimuli on sound identification, the following were used: the identification rates of sounds for which the visual stimulus denoted its sound source, the rates of other sounds for which the visual stimulus did not denote the sound source, and the frequency of false hearing of a sound that was not presented for each sound set. Results of the four experiments demonstrated that a picture or a written word promoted identification of the sound when it was related to the sound, particularly when the visual stimulus was presented for 5 s simultaneously with the sounds. However, a visual stimulus preceding the sounds had a benefit only for the picture, not for the written word. Furthermore, presentation with a picture denoting a sound simultaneously with the sound reduced the frequency of false hearing. These results suggest three ways that presenting a visual stimulus affects identification of the auditory stimulus. First, activation of the visual representation extracted directly from the picture promotes identification of the denoted sound and suppresses the processing of sounds for which the visual stimulus did not denote the sound source. Second, effects based on processing of the conceptual information promote identification of the denoted sound and suppress the processing of sounds for which the visual stimulus did not denote the sound source. Third, processing of the concurrent visual representation suppresses false hearing.
authors.affiliation: Faculty of Media Theories and Production, Aichi Shukutoku UniversityAichi, Japan; Faculty of Engineering, Yamaguchi UniversityYamaguchi, Japan.
authors.forename: Yuko
authors.initials: Y
authors.lastname: Masakura
authors.affiliation: Faculty of Engineering, Yamaguchi UniversityYamaguchi, Japan; Department of Psychology, Chiba UniversityChiba, Japan.
authors.forename: Makoto
authors.initials: M
authors.lastname: Ichikawa
authors.affiliation: Faculty of Marine Technology, Tokyo University of Marine Science and Technology Tokyo, Japan.
authors.forename: Koichi
authors.initials: K
authors.lastname: Shimono
authors.affiliation: Faculty of Engineering, Yamaguchi University Yamaguchi, Japan.
authors.forename: Reio
authors.initials: R
authors.lastname: Nakatsuka
keywords: false hearing; mixed sounds; visual representation; visual stimulus content; visual stimulus timing