Cross-modal links among vision, audition, and touch in complex environments.
OBJECTIVE
This study sought to determine whether performance effects of cross-modal spatial links that were observed in earlier laboratory studies scale to more complex environments and need to be considered in multimodal interface design. It also revisits the unresolved issue of cross-modal cuing asymmetries.


BACKGROUND
Previous laboratory studies employing simple cues, tasks, and/or targets have demonstrated that the efficiency of processing visual, auditory, and tactile stimuli is affected by the modality, lateralization, and timing of surrounding cues. Very few studies have investigated these cross-modal constraints in the context of more complex environments to determine whether they scale and how complexity affects the nature of cross-modal cuing asymmetries.


METHODS
Amicroworld simulation of battlefield operations with a complex task set and meaningful visual, auditory, and tactile stimuli was used to investigate cuing effects for all cross-modal pairings.


RESULTS
Significant asymmetric performance effects of cross-modal spatial links were observed. Auditory cues shortened response latencies for collocated visual targets but visual cues did not do the same for collocated auditory targets. Responses to contralateral (rather than ipsilateral) targets were faster for tactually cued auditory targets and each visual-tactile cue-target combination, suggesting an inhibition-of-return effect.


CONCLUSIONS
The spatial relationships between multimodal cues and targets significantly affect target response times in complex environments. The performance effects of cross-modal links and the observed cross-modal cuing asymmetries need to be examined in more detail and considered in future interface design.


CONCLUSIONS
The findings from this study have implications for the design of multimodal and adaptive interfaces and for supporting attention management in complex, data-rich domains.
Ferris
Thomas K
TK
Department of Industrial and Operations Engineering, Center for Ergonomics, University of Michigan, 1205 Beal Ave., Ann Arbor, MI 48109, USA.
Sarter
Nadine B
NB
D001307:Auditory Perception; D003463:Cues; D006801:Humans; D008889:Military Personnel; D011597:Psychomotor Performance; D011930:Reaction Time; D014110:Touch; D014481:United States; D014584:User-Computer Interface; D014796:Visual Perception