Towards unified brain-to-text decoding across speech production and perception
A new AI framework decodes Mandarin sentences from brain signals for both speaking and listening.
A research team from multiple Chinese institutions has published a groundbreaking paper titled 'Towards unified brain-to-text decoding across speech production and perception.' The work introduces a novel AI framework capable of translating brain activity into coherent Mandarin sentences, a significant leap for non-alphabetic languages. Unlike previous studies focused on single modalities like imagined speech, this model works for both speech production (when a person thinks about speaking) and speech perception (when they listen). The system first decodes neural signals into the basic phonetic components of Mandarin, known as Pinyin initials and finals. These components are then fed into a specially post-trained 7-billion-parameter large language model (LLM) that maps the sequence of toneless Pinyin into grammatically correct Chinese text.
Remarkably, the team's optimized LLM framework outperforms commercial models with hundreds of billions of parameters, demonstrating efficient, specialized design. The research also provided direct neural comparisons between the two modalities, revealing that speech production activates broader brain regions than perception and that shared neural channels show similar patterns with a perceivable time delay. This work establishes the first unified decoding framework for a logosyllabic language like Mandarin and paves the way for future brain-computer interfaces that can operate across multiple communication modes, from silent thought to auditory processing.
- Unified framework decodes both speech production (thinking) and perception (listening) from brain signals in Mandarin.
- Uses a post-trained 7B-parameter LLM to map Pinyin to text, outperforming much larger commercial models.
- Revealed key neural insights: production uses broader brain regions, and perception shows a temporal delay versus production.
Why It Matters
This advances non-invasive brain-computer interfaces for communication, especially for patients with speech impairments, and deepens our understanding of how the brain processes language.