title: Auditory scene analysis and sonified visual images. Does consonance negatively impact on object formation when using complex sonified stimuli?
abstract: A critical task for the brain is the sensory representation and identification of perceptual objects in the world. When the visual sense is impaired, hearing and touch must take primary roles and in recent times compensatory techniques have been developed that employ the tactile or auditory system as a substitute for the visual system. Visual-to-auditory sonifications provide a complex, feature-based auditory representation that must be decoded and integrated into an object-based representation by the listener. However, we don't yet know what role the auditory system plays in the object integration stage and whether the principles of auditory scene analysis apply. Here we used coarse sonified images in a two-tone discrimination task to test whether auditory feature-based representations of visual objects would be confounded when their features conflicted with the principles of auditory consonance. We found that listeners (N = 36) performed worse in an object recognition task when the auditory feature-based representation was harmonically consonant. We also found that this conflict was not negated with the provision of congruent audio-visual information. The findings suggest that early auditory processes of harmonic grouping dominate the object formation process and that the complexity of the signal, and additional sensory information have limited effect on this.
authors.affiliation: Crossmodal Cognition Lab, Department of Psychology, University of Bath Bath, UK ; Biological and Experimental Psychology Group, School of Biological and Chemical Sciences, Queen Mary University of London London, UK.
authors.forename: David J
authors.initials: DJ
authors.lastname: Brown
authors.affiliation: Centre for Vision, Speech and Signal Processing, University of Surrey Guildford, UK.
authors.forename: Andrew J R
authors.initials: AJ
authors.lastname: Simpson
authors.affiliation: Crossmodal Cognition Lab, Department of Psychology, University of Bath Bath, UK.
authors.forename: Michael J
authors.initials: MJ
authors.lastname: Proulx
keywords: auditory scene analysis; blindness; consonance; cross-modal; sensory substitution; signal complexity