New AI Speech Trick Makes Voice Assistants Sound More Natural

Speech language models rely on discrete speech representations from pretrained codecs, but those codecs are optimized for compression and don't account for the autoregressive nature of language model training—which can limit performance. This paper proposes a new framework that explicitly aligns speech tokenization with autoregressive training by introducing autoregressive-compatible constraints that encourage temporal consistency and predictability in token sequences. It also uses a heterogeneous downsampling strategy across different layers of speech tokens, distinguishing semantic from acoustic layers to better align semantic tokens with text. Experiments across multiple benchmarks show the approach narrows the gap between speech compression and generative modeling, supports more effective continued pretraining of existing language models on speech data, and consistently improves performance across multiple codecs.