MSR-Codec: A Low-Bitrate Multi-Stream Residual Codec for High-Fidelity Speech Generation with Information Disentanglement

A research team has introduced MSR-Codec, a novel audio codec architecture that fundamentally changes how speech is encoded for generation systems. Unlike traditional codecs that compress audio into a single stream, MSR-Codec separates speech into four distinct components: semantic content, speaker timbre, prosody (rhythm and intonation), and residual details. This multi-stream approach with information disentanglement enables high-fidelity reconstruction at remarkably low bitrates while maintaining competitive performance metrics. The codec serves as the foundation for a two-stage language model designed for text-to-speech synthesis.

The technical breakthrough lies in the codec's ability to independently manipulate these speech components, which has significant implications for voice AI applications. Researchers demonstrated that their lightweight TTS system, built using MSR-Codec, achieves state-of-the-art Word Error Rate and superior speaker similarity compared to larger models, despite requiring minimal training data. This efficiency makes the technology particularly valuable for voice conversion tasks, where users can modify a speaker's timbre while preserving the original prosody, or vice versa. The team has made their inference code, pre-trained models, and audio samples publicly available, potentially accelerating development in speech synthesis and voice cloning technologies.