PRIME-Speech lets S2T LLMs generate speech without losing accuracy
Fine-tuning for speech output often kills perception; this method doesn't.
Deep Dive
PRIME-Speech is a framework that adds speech-to-speech generation to existing speech-to-text LLMs without degrading their original understanding. It uses a frozen backbone with a causal audio post-decoder that reads intermediate hidden states, enabling low-latency, low-WER spoken responses across translation, QA, and multi-turn dialogue. Multi-token prediction reduces the effective codec prediction rate and improves first-audio latency.
Key Points
- Frozen backbone preserves original S2T performance; only speech-generation modules are trained.
- Causal audio post-decoder reads hidden states directly, avoiding a serial text-to-speech bottleneck.
- Multi-token prediction reduces codec prediction rate by 2–3x and improves first-audio latency.
Why It Matters
Enables true speech-to-speech LLMs without sacrificing understanding—critical for real-time voice assistants and dialogue systems.