title: Hybrid fNIRS-EEG based classification of auditory and visual perception processes.
abstract: For multimodal Human-Computer Interaction (HCI), it is very useful to identify the modalities on which the user is currently processing information. This would enable a system to select complementary output modalities to reduce the user's workload. In this paper, we develop a hybrid Brain-Computer Interface (BCI) which uses Electroencephalography (EEG) and functional Near Infrared Spectroscopy (fNIRS) to discriminate and detect visual and auditory stimulus processing. We describe the experimental setup we used for collection of our data corpus with 12 subjects. On this data, we performed cross-validation evaluation, of which we report accuracy for different classification conditions. The results show that the subject-dependent systems achieved a classification accuracy of 97.8% for discriminating visual and auditory perception processes from each other and a classification accuracy of up to 94.8% for detecting modality-specific processes independently of other cognitive activity. The same classification conditions could also be discriminated in a subject-independent fashion with accuracy of up to 94.6 and 86.7%, respectively. We also look at the contributions of the two signal types and show that the fusion of classifiers using different features significantly increases accuracy.
authors.affiliation: Cognitive Systems Lab, Institute of Anthropomatics and Robotics, Karlsruhe Institute of Technology Karlsruhe, Germany.
authors.forename: Felix
authors.initials: F
authors.lastname: Putze
authors.affiliation: Cognitive Systems Lab, Institute of Anthropomatics and Robotics, Karlsruhe Institute of Technology Karlsruhe, Germany.
authors.forename: Sebastian
authors.initials: S
authors.lastname: Hesslinger
authors.affiliation: Department of Psychology, Center for Cognition and Brain Studies, The Chinese University of Hong Kong Hong Kong, China ; Temasek Laboratories, National University of Singapore Singapore, Singapore.
authors.forename: Chun-Yu
authors.initials: CY
authors.lastname: Tse
authors.affiliation: Nuffield Department of Clinical Neurosciences, John Radcliffe Hospital Oxford, UK.
authors.forename: YunYing
authors.initials: Y
authors.lastname: Huang
authors.affiliation: Cognitive Systems Lab, Institute of Anthropomatics and Robotics, Karlsruhe Institute of Technology Karlsruhe, Germany.
authors.forename: Christian
authors.initials: C
authors.lastname: Herff
authors.affiliation: Institute for Infocomm Research (I2R), ASTAR Singapore, Singapore.
authors.forename: Cuntai
authors.initials: C
authors.lastname: Guan
authors.affiliation: Cognitive Systems Lab, Institute of Anthropomatics and Robotics, Karlsruhe Institute of Technology Karlsruhe, Germany.
authors.forename: Tanja
authors.initials: T
authors.lastname: Schultz
keywords: EEG; brain-computer interface; fNIRS; visual and auditory perception