Real-time decoding of question-and-answer speech dialogue using human cortical activity

Publication Title
Nature Communications
Publication/Creation Date
July 30 2019
University Of California, San Francisco (creator)
David Moses (creator)
Edward Chang (creator)
Matthew K. Leonard (creator)
Joseph G. Makin (creator)
Facebook (contributor)
Media Type
Journal Article
Persuasive Intent

Natural communication often occurs in dialogue, differentially engaging auditory and sensorimotor brain regions during listening and speaking. However, previous attempts to decode speech directly from the human brain typically consider listening or speaking tasks in isolation. Here, human participants listened to questions and responded aloud with answers while we used high-density electrocorticography (ECoG) recordings to detect when they heard or said an utterance and to then decode the utterance’s identity. Because certain answers were only plausible responses to certain questions, we could dynamically update the prior probabilities of each answer using the decoded question likelihoods as context. We decode produced and perceived utterances with accuracy rates as high as 61% and 76%, respectively (chance is 7% and 20%). Contextual integration of decoded question likelihoods significantly improves answer decoding. These results demonstrate real-time decoding of speech in an interactive, conversational setting, which has important implications for patients who are unable to communicate.

This research was funded by Facebook.
HCI Platform
Discursive Type
Location on Body
Marketing Keywords
Facebook Reality Labs

Date archived
August 5 2019
Last edited
February 28 2020
How to cite this entry
University of California, San Francisco, David Moses, Edward Chang, Matthew K. Leonard, Joseph G. Makin. (July 30 2019). "Real-time decoding of question-and-answer speech dialogue using human cortical activity". Nature Communications. Springer Nature Publishing. Fabric of Digital Life.