title: A hybrid BCI based on EEG and fNIRS signals improves the performance of decoding motor imagery of both force and speed of hand clenching.
abstract: OBJECTIVE
  In order to increase the number of states classified by a brain-computer interface (BCI), we utilized a motor imagery task where subjects imagined both force and speed of hand clenching.
  
  
  METHODS
  The BCI utilized simultaneously recorded electroencephalographic (EEG) and functional near-infrared spectroscopy (fNIRS) signals. The time-phase-frequency feature was extracted from EEG, whereas the HbD [the difference of oxy-hemoglobin (HbO) and deoxy-hemoglobin (Hb)] feature was used to improve the classification accuracy of fNIRS. The EEG and fNIRS features were combined and optimized using the joint mutual information (JMI) feature selection criterion; then the extracted features were classified with the extreme learning machines (ELMs).
  
  
  RESULTS
  In this study, the averaged classification accuracy of EEG signals achieved by the time-phase-frequency feature improved by 7%, to 18%, more than the single-type feature, and improved by 15% more than common spatial pattern (CSP) feature. The HbD feature of fNIRS signals improved the accuracy by 1%, to 4%, more than Hb, HbO, or HbT (total hemoglobin). The EEG-fNIRS feature for decoding motor imagery of both force and speed of hand clenching achieved an accuracy of 89% ± 2%, and improved the accuracy by 1% to 5% more than the sole EEG or fNIRS feature.
  
  
  CONCLUSIONS
  Our novel motor imagery paradigm improves BCI performance by increasing the number of extracted commands. Both the time-phase-frequency and the HbD feature improve the classification accuracy of EEG and fNIRS signals, respectively, and the hybrid EEG-fNIRS technique achieves a higher decoding accuracy for two-class motor imagery, which may provide the framework for future multi-modal online BCI systems.
authors.affiliation: State Key Laboratory of Robotics, Shenyang Institute of Automation (SIA), Chinese Academy of Sciences (CAS), Shenyang 110016, People's Republic of China. University of Chinese Academy of Sciences, Beijing 100049, People's Republic of China.
authors.forename: Xuxian
authors.initials: X
authors.lastname: Yin
authors.forename: Baolei
authors.initials: B
authors.lastname: Xu
authors.forename: Changhao
authors.initials: C
authors.lastname: Jiang
authors.forename: Yunfa
authors.initials: Y
authors.lastname: Fu
authors.forename: Zhidong
authors.initials: Z
authors.lastname: Wang
authors.forename: Hongyi
authors.initials: H
authors.lastname: Li
authors.forename: Gang
authors.initials: G
authors.lastname: Shi
mesh_terms: D000328:Adult; D001931:Brain Mapping / Q000379:methods; D062207:Brain-Computer Interfaces; D004569:Electroencephalography / Q000379:methods*; D005260:Female; D018737:Hand Strength / Q000502:physiology*; D006801:Humans; D007092:Imagination / Q000502:physiology*; D008297:Male; D009044:Motor Cortex / Q000502:physiology*; D064847:Multimodal Imaging / Q000379:methods; D011597:Psychomotor Performance / Q000502:physiology; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D019265:Spectroscopy, Near-Infrared / Q000379:methods*; D013314:Stress, Mechanical; D018511:Systems Integration