title: Unsupervised Decoding of Long-Term, Naturalistic Human Neural Recordings with Automated Video and Audio Annotations.
abstract: Fully automated decoding of human activities and intentions from direct neural recordings is a tantalizing challenge in brain-computer interfacing. Implementing Brain Computer Interfaces (BCIs) outside carefully controlled experiments in laboratory settings requires adaptive and scalable strategies with minimal supervision. Here we describe an unsupervised approach to decoding neural states from naturalistic human brain recordings. We analyzed continuous, long-term electrocorticography (ECoG) data recorded over many days from the brain of subjects in a hospital room, with simultaneous audio and video recordings. We discovered coherent clusters in high-dimensional ECoG recordings using hierarchical clustering and automatically annotated them using speech and movement labels extracted from audio and video. To our knowledge, this represents the first time techniques from computer vision and speech processing have been used for natural ECoG decoding. Interpretable behaviors were decoded from ECoG data, including moving, speaking and resting; the results were assessed by comparison with manual annotation. Discovered clusters were projected back onto the brain revealing features consistent with known functional areas, opening the door to automated functional brain mapping in natural settings.
authors.affiliation: Department of Computer Science and Engineering, University of WashingtonSeattle, WA, USA; Institute for Neuroengineering, University of WashingtonSeattle, WA, USA; eScience Institute, University of WashingtonSeattle, WA, USA; Center for Sensorimotor Neural Engineering, University of WashingtonSeattle, WA, USA.
authors.forename: Nancy X R
authors.initials: NX
authors.lastname: Wang
authors.affiliation: Center for Sensorimotor Neural Engineering, University of WashingtonSeattle, WA, USA; Department of Rehabilitation Medicine, University of WashingtonSeattle, WA, USA.
authors.forename: Jared D
authors.initials: JD
authors.lastname: Olson
authors.affiliation: Institute for Neuroengineering, University of WashingtonSeattle, WA, USA; Center for Sensorimotor Neural Engineering, University of WashingtonSeattle, WA, USA; Department of Neurological Surgery, University of WashingtonSeattle, WA, USA.
authors.forename: Jeffrey G
authors.initials: JG
authors.lastname: Ojemann
authors.affiliation: Department of Computer Science and Engineering, University of WashingtonSeattle, WA, USA; Institute for Neuroengineering, University of WashingtonSeattle, WA, USA; Center for Sensorimotor Neural Engineering, University of WashingtonSeattle, WA, USA.
authors.forename: Rajesh P N
authors.initials: RP
authors.lastname: Rao
authors.affiliation: Institute for Neuroengineering, University of WashingtonSeattle, WA, USA; eScience Institute, University of WashingtonSeattle, WA, USA; Department of Biology, University of WashingtonSeattle, WA, USA.
authors.forename: Bingni W
authors.initials: BW
authors.lastname: Brunton
keywords: automation; computer vision; electrocorticography (ECoG); functional brain mapping; long-term recording; neural decoding; speech processing; unsupervised machine learning