Multisensory integration with a head-mounted display: background visual motion and sound motion.
OBJECTIVE
The aim of this study was to assess how background visual motion and the relative movement of sound affect a head-mounted display (HMD) wearer's performance at a task requiring integration of auditory and visual information.


BACKGROUND
HMD users are often mobile. A commercially available speaker in a fixed location delivers auditory information affordably to the HMD user. However, previous research has shown that mobile HMD users perform poorly at tasks that require integration of visual and auditory information when sound comes from a free-field speaker. The specific cause of the poor task performance is unknown.


METHODS
Participants counted audiovisual events that required integration of sounds delivered via a free-field speaker and vision on an HMD. Participants completed the task while either walking around a room, sitting in the room, or sitting inside a mobile room that allowed separate manipulation of background visual motion and speaker motion.


RESULTS
Participants' accuracy at counting target audiovisual events was worse when participants were walking than when sitting at a desk, p = .032. Compared with when they were sitting at a desk, participants' accuracy at counting target audiovisual events showed a trend to be worse when they experienced a combination of background visual motion and the relative movement of sound, p = .058.


CONCLUSIONS
Multisensory integration performance is least effective when HMD users experience a combination of background visual motion and relative movement of sound. Eye reflexes may play an important role.


CONCLUSIONS
Results apply to situations in which HMD wearers are mobile when receiving multimodal information, as in health care and military contexts.
Harrison
William J
WJ
The University of Queensland, St Lucia, Queensland, Australia.
Thompson
Matthew B
MB
Sanderson
Penelope M
PM
D000293:Adolescent; D000328:Adult; D003626:Data Display; D005260:Female; D006257:Head; D006801:Humans; D008297:Male; D009038:Motion; D013016:Sound; D013647:Task Performance and Analysis; D014796:Visual Perception; D055815:Young Adult