title: Evaluating true BCI communication rate through mutual information and language models.
abstract: Brain-computer interface (BCI) systems are a promising means for restoring communication to patients suffering from "locked-in" syndrome. Research to improve system performance primarily focuses on means to overcome the low signal to noise ratio of electroencephalogric (EEG) recordings. However, the literature and methods are difficult to compare due to the array of evaluation metrics and assumptions underlying them, including that: 1) all characters are equally probable, 2) character selection is memoryless, and 3) errors occur completely at random. The standardization of evaluation metrics that more accurately reflect the amount of information contained in BCI language output is critical to make progress. We present a mutual information-based metric that incorporates prior information and a model of systematic errors. The parameters of a system used in one study were re-optimized, showing that the metric used in optimization significantly affects the parameter values chosen and the resulting system performance. The results of 11 BCI communication studies were then evaluated using different metrics, including those previously used in BCI literature and the newly advocated metric. Six studies' results varied based on the metric used for evaluation and the proposed metric produced results that differed from those originally published in two of the studies. Standardizing metrics to accurately reflect the rate of information transmission is critical to properly evaluate and compare BCI communication systems and advance the field in an unbiased manner.
authors.affiliation: Department of Bioengineering, University of California Los Angeles, Los Angeles, California, United States of America ; Medical Imaging Informatics Group, University of California Los Angeles, Los Angeles, California, United States of America.
authors.forename: William
authors.initials: W
authors.lastname: Speier
authors.forename: Corey
authors.initials: C
authors.lastname: Arnold
authors.forename: Nader
authors.initials: N
authors.lastname: Pouratian
mesh_terms: D062207:Brain-Computer Interfaces; D006801:Humans; D008962:Models, Theoretical; D011381:Programming Languages; D011782:Quadriplegia