title: Exploring combinations of auditory and visual stimuli for gaze-independent brain-computer interfaces.
abstract: For Brain-Computer Interface (BCI) systems that are designed for users with severe impairments of the oculomotor system, an appropriate mode of presenting stimuli to the user is crucial. To investigate whether multi-sensory integration can be exploited in the gaze-independent event-related potentials (ERP) speller and to enhance BCI performance, we designed a visual-auditory speller. We investigate the possibility to enhance stimulus presentation by combining visual and auditory stimuli within gaze-independent spellers. In this study with N = 15 healthy users, two different ways of combining the two sensory modalities are proposed: simultaneous redundant streams (Combined-Speller) and interleaved independent streams (Parallel-Speller). Unimodal stimuli were applied as control conditions. The workload, ERP components, classification accuracy and resulting spelling speed were analyzed for each condition. The Combined-speller showed a lower workload than uni-modal paradigms, without the sacrifice of spelling performance. Besides, shorter latencies, lower amplitudes, as well as a shift of the temporal and spatial distribution of discriminative information were observed for Combined-speller. These results are important and are inspirations for future studies to search the reason for these differences. For the more innovative and demanding Parallel-Speller, where the auditory and visual domains are independent from each other, a proof of concept was obtained: fifteen users could spell online with a mean accuracy of 87.7% (chance level <3%) showing a competitive average speed of 1.65 symbols per minute. The fact that it requires only one selection period per symbol makes it a good candidate for a fast communication channel. It brings a new insight into the true multisensory stimuli paradigms. Novel approaches for combining two sensory modalities were designed here, which are valuable for the development of ERP-based BCI paradigms.
authors.affiliation: Department of Biomedical Engineering, Tianjin University, Tianjin, China; Neurotechnology Group, Berlin Institute of Technology, Berlin, Germany.
authors.forename: Xingwei
authors.initials: X
authors.lastname: An
authors.affiliation: Neurotechnology Group, Berlin Institute of Technology, Berlin, Germany; Machine Learning Group, Berlin Institute of Technology, Berlin, Germany.
authors.forename: Johannes
authors.initials: J
authors.lastname: Höhne
authors.affiliation: Department of Biomedical Engineering, Tianjin University, Tianjin, China.
authors.forename: Dong
authors.initials: D
authors.lastname: Ming
authors.affiliation: Neurotechnology Group, Berlin Institute of Technology, Berlin, Germany.
authors.forename: Benjamin
authors.initials: B
authors.lastname: Blankertz
mesh_terms: D000161:Acoustic Stimulation; D000328:Adult; D000704:Analysis of Variance; D001519:Behavior; D062207:Brain-Computer Interfaces; D003657:Decision Making; D004569:Electroencephalography; D005071:Evoked Potentials; D005133:Eye Movements; D005260:Female; D006801:Humans; D008297:Male; D010775:Photic Stimulation; D015203:Reproducibility of Results; D016896:Treatment Outcome