Multisensory integration with a head-mounted display: sound delivery and self-motion.
OBJECTIVE
We tested whether the method of sound delivery affects people's ability to integrate information from multiple modalities when they are walking and using a head-mounted display (HMD).


BACKGROUND
HMDs increasingly support mobile work. Human operators may benefit from auditory support when using an HMD. However, it is unclear whether sound is better delivered publicly in free field or privately via earpiece and what the effect of walking is.


METHODS
Participants identified mismatches between sounds and visual information on an HMD. Participants heard the sounds via either earpiece or free field while they either sat or walked about the test room.


RESULTS
When using an earpiece, participants performed the mismatch task equally well whether sitting or walking, but when using free-field sound, participants performed the task significantly worse when walking than when sitting (p = .006).


CONCLUSIONS
The worse performance for participants using free-field sound while walking may relate to spatial and motion inconsistencies between visual events on the head-referenced HMD and auditory events from world-referenced speakers. Researchers should more frequently examine the effect of self-motion on people's ability to perform various multisensory tasks.


CONCLUSIONS
When multisensory integration tasks are performed with an HMD and free-field delivery of sound, as may happen in medicine, transportation, or industry, performance may suffer when the relative location of sound changes as the user moves.
Thompson
Matthew B
MB
ARC Key Centre for Human Factors, The University of Queensland, St Lucia, Queensland, Australia.
Sanderson
Penelope M
PM
D000293:Adolescent; D000328:Adult; D001307:Auditory Perception; D003626:Data Display; D004867:Equipment Design; D006801:Humans; D019212:Multimedia; D011597:Psychomotor Performance; D014584:User-Computer Interface; D014796:Visual Perception; D016138:Walking; D055815:Young Adult