MoDAl: Self-Supervised Neural Modality Discovery via Decorrelation for Speech Neuroprosthesis
New self-supervised method decodes brain signals with 21.6% error rate using Broca's area
MoDAl (Modality Decorrelation and Alignment) tackles a key limitation in speech neuroprosthesis: current systems decode only from motor cortical areas, ignoring regions like Broca's area (area 44) that carry complementary linguistic information. The framework uses a dual-objective training scheme. First, a contrastive loss aligns parallel brain encoders with text embeddings from a pretrained large language model (LLM). Second, a decorrelation loss prevents those encoders from learning redundant representations—forcing them to discover diverse neurolinguistic modalities.
On the Brain-to-Text Benchmark '24, MoDAl reduced word error rate (WER) from 26.3% (previous best end-to-end method) to 21.6%—a relative improvement of nearly 18%. Critically, the gain from incorporating area 45 signals came entirely from the decorrelation mechanism. Analysis shows the discovered modalities have functional specialization: encoders receiving area 44 input capture structural and syntactic features (sentence length, grammatical voice, wh-words), consistent with Broca's area's known role. This breakthrough could dramatically improve communication for individuals with speech-impairing conditions by leveraging more of the brain's linguistic architecture.
- MoDAl reduces WER from 26.3% to 21.6% on the Brain-to-Text Benchmark '24, an 18% relative improvement.
- The decorrelation loss forces parallel brain encoders to learn diverse representations, enabling discovery of modality specialization.
- Area 44 (Broca's area) encodes syntactic properties like sentence length and grammatical voice, confirmed by MoDAl's analysis.
Why It Matters
Leveraging additional brain regions boosts speech decoding accuracy, bringing brain-computer interfaces closer to restoring fluent communication.