Beyond p-values in the evaluation of brain-computer interfaces: A Bayesian estimation approach.
To statistically evaluate the performance of brain-computer interfaces (BCIs), researchers usually rely on null hypothesis significance testing (NHST), i.e. p-values. However, over-reliance on NHST is often identified as one of the causes of the recent reproducibility crisis in psychology and neuroscience.



In this paper we propose Bayesian estimation as an alternative to NHST in the analysis of BCI performance data. For the three most common experimental designs in BCI research - which would usually be analyzed using a t-test, a linear regression, or an ANOVA - we develop hierarchical models and estimate their parameters using Bayesian inference. Furthermore, we show that the described models are special cases of the hierarchical generalized linear model (HGLM), which we propose as a general framework for the analysis of BCI performance.



We demonstrate the effectiveness of the proposed models on three real datasets and show how the results obtained with Bayesian estimation can give a nuanced insight into BCI performance data. Additionally, we provide all the data and code necessary to reproduce the presented results.



Compared to NHST, Bayesian estimation with the HGLM allows more flexibility in the analysis of BCI performance data from nested experimental designs, and the obtained results have a more straightforward interpretation.



Besides gains in flexibility and interpretability, a wider adoption of the Bayesian estimation approach in BCI studies could bring about greater transparency in data analysis, allow accumulation of knowledge across studies, and reduce questionable practices such as "p-hacking".
Bit&Brain Technologies S.L., Paseo de Sagasta 19, 50008 Zaragoza, Spain. Electronic address: filip.melinscak@bitbrain.es.
Filip
F
Melinscak
Bit&Brain Technologies S.L., Paseo de Sagasta 19, 50008 Zaragoza, Spain; University of Zaragoza, Aragon Institute of Engineering Research (I3A), I+D+i Building, Mariano Esquillor s/n, 50018 Zaragoza, Spain.
Luis
L
Montesano
D000513:Alpha Rhythm; D001499:Bayes Theorem; D062207:Brain-Computer Interfaces; D003627:Data Interpretation, Statistical; D004569:Electroencephalography / Q000379:methods; D005069:Evaluation Studies as Topic; D005074:Evoked Potentials, Visual; D006801:Humans; D007092:Imagination / Q000502:physiology; D017705:Lignans; D016014:Linear Models; D055641:Mathematical Concepts; D009043:Motor Activity / Q000502:physiology; D009044:Motor Cortex / Q000502:physiology; D009146:Music; D011340:Problem Solving / Q000502:physiology; D012146:Rest; D014796:Visual Perception / Q000502:physiology
D017705:Lignans; C560138:ananolignan F
Bayesian estimation; Brain–computer interface (BCI); Classification accuracy; Generalized linear model (GLM); Hierarchical modeling; Null hypothesis significance testing (NHST)