ReLMCodec cracks speech token prediction for 650bps audio
New speech codec bridges predictability and fidelity at ultra-low bitrates
Deep Dive
The authors introduce ReLMCodec, a low-bitrate single-codebook speech codec that makes speech tokens easier for autoregressive models to predict while keeping acoustic detail for high-fidelity reconstruction. Their analysis shows clearer phoneme structure before discrete code assignment is associated with easier autoregressive prediction, but phoneme structure alone isn’t enough for reconstruction. ReLMCodec follows a preserve–control
Key Points
- ReLMCodec operates at 650–800 bps with a single-codebook design, outperforming prior low-bitrate speech codecs.
- It preserves phoneme structure via frozen SSL features and refines tokens using a WavLM-Large L24 teacher.
- Downstream TTS benefits include +intelligibility and +speaker similarity in evaluations.
Why It Matters
Could unlock real-time, ultra-low-bandwidth TTS and voice AI with near-lossless clarity.