title: Performance assessment in brain-computer interface-based augmentative and alternative communication.
abstract: A large number of incommensurable metrics are currently used to report the performance of brain-computer interfaces (BCI) used for augmentative and alterative communication (AAC). The lack of standard metrics precludes the comparison of different BCI-based AAC systems, hindering rapid growth and development of this technology. This paper presents a review of the metrics that have been used to report performance of BCIs used for AAC from January 2005 to January 2012. We distinguish between Level 1 metrics used to report performance at the output of the BCI Control Module, which translates brain signals into logical control output, and Level 2 metrics at the Selection Enhancement Module, which translates logical control to semantic control. We recommend that: (1) the commensurate metrics Mutual Information or Information Transfer Rate (ITR) be used to report Level 1 BCI performance, as these metrics represent information throughput, which is of interest in BCIs for AAC; 2) the BCI-Utility metric be used to report Level 2 BCI performance, as it is capable of handling all current methods of improving BCI performance; (3) these metrics should be supplemented by information specific to each unique BCI configuration; and (4) studies involving Selection Enhancement Modules should report performance at both Level 1 and Level 2 in the BCI system. Following these recommendations will enable efficient comparison between both BCI Control and Selection Enhancement Modules, accelerating research and development of BCI-based AAC systems.
authors.affiliation: Department of Biomedical Engineering, University of Michigan, Ann Arbor, MI, USA.
authors.forename: David E
authors.initials: DE
authors.lastname: Thompson
authors.forename: Stefanie
authors.initials: S
authors.lastname: Blain-Moraes
authors.forename: Jane E
authors.initials: JE
authors.lastname: Huggins
mesh_terms: D062207:Brain-Computer Interfaces; D003143:Communication Devices for People with Disabilities; D006801:Humans; D015203:Reproducibility of Results; D012815:Signal Processing, Computer-Assisted