title: Decoding spectrotemporal features of overt and covert speech from the human cortex.
abstract: Auditory perception and auditory imagery have been shown to activate overlapping brain regions. We hypothesized that these phenomena also share a common underlying neural representation. To assess this, we used electrocorticography intracranial recordings from epileptic patients performing an out loud or a silent reading task. In these tasks, short stories scrolled across a video screen in two conditions: subjects read the same stories both aloud (overt) and silently (covert). In a control condition the subject remained in a resting state. We first built a high gamma (70-150 Hz) neural decoding model to reconstruct spectrotemporal auditory features of self-generated overt speech. We then evaluated whether this same model could reconstruct auditory speech features in the covert speech condition. Two speech models were tested: a spectrogram and a modulation-based feature space. For the overt condition, reconstruction accuracy was evaluated as the correlation between original and predicted speech features, and was significant in each subject (p < 10(-5); paired two-sample t-test). For the covert speech condition, dynamic time warping was first used to realign the covert speech reconstruction with the corresponding original speech from the overt condition. Reconstruction accuracy was then evaluated as the correlation between original and reconstructed speech features. Covert reconstruction accuracy was compared to the accuracy obtained from reconstructions in the baseline control condition. Reconstruction accuracy for the covert condition was significantly better than for the control condition (p < 0.005; paired two-sample t-test). The superior temporal gyrus, pre- and post-central gyrus provided the highest reconstruction information. The relationship between overt and covert speech reconstruction depended on anatomy. These results provide evidence that auditory representations of covert speech can be reconstructed from models that are built from an overt speech data set, supporting a partially shared neural substrate.
authors.affiliation: Helen Wills Neuroscience Institute, University of California Berkeley, CA, USA ; Department of Bioengineering, École Polytechnique Fédérale de Lausanne Lausanne, Switzerland.
authors.forename: Stéphanie
authors.initials: S
authors.lastname: Martin
authors.affiliation: New York State Department of Health, Wadsworth Center Albany, NY, USA ; Department of Neurology, Albany Medical College Albany, NY, USA.
authors.forename: Peter
authors.initials: P
authors.lastname: Brunner
authors.affiliation: Helen Wills Neuroscience Institute, University of California Berkeley, CA, USA.
authors.forename: Chris
authors.initials: C
authors.lastname: Holdgraf
authors.affiliation: Department of Neurology, Otto-von-Guericke-Universitat Magdeburg, Germany.
authors.forename: Hans-Jochen
authors.initials: HJ
authors.lastname: Heinze
authors.affiliation: Department of Neurology, Johns Hopkins University School of Medicine Baltimore, MD, USA.
authors.forename: Nathan E
authors.initials: NE
authors.lastname: Crone
authors.affiliation: Helen Wills Neuroscience Institute, University of California Berkeley, CA, USA ; Applied Neurocognitive Psychology, Carl-von-Ossietzky University Oldenburg, Germany.
authors.forename: Jochem
authors.initials: J
authors.lastname: Rieger
authors.affiliation: New York State Department of Health, Wadsworth Center Albany, NY, USA ; Department of Neurology, Albany Medical College Albany, NY, USA.
authors.forename: Gerwin
authors.initials: G
authors.lastname: Schalk
authors.affiliation: Helen Wills Neuroscience Institute, University of California Berkeley, CA, USA ; Department of Psychology, University of California Berkeley, CA, USA.
authors.forename: Robert T
authors.initials: RT
authors.lastname: Knight
authors.affiliation: Helen Wills Neuroscience Institute, University of California Berkeley, CA, USA.
authors.forename: Brian N
authors.initials: BN
authors.lastname: Pasley
keywords: covert speech; decoding model; electrocorticography; pattern recognition; speech production