VoiceChat-TTS enables real-time speech synthesis with seamless barge-in
Supports mid-utterance interruptions without resetting the KV cache for true duplex conversation
Spoken dialogue is a natural interface for AI agents, but most speech language models still rely on turn-based operation, forcing users to wait for the system to finish before speaking. Two-way duplex models have tried to cut latency by replacing multi-stage pipelines, but they often sacrifice speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized together. VoiceChat-TTS, a new paper on arXiv, breaks this trade-off by introducing a low-latency, continuous, and streamable text-to-speech model specifically built for interactive agents.
The key innovation is that VoiceChat-TTS is driven directly by LLM text-token streams rather than requiring a separate ASR-then-TTS cascade. It uses explicit control tokens to support user barge-in, so the model can be interrupted mid-utterance and immediately adapt—without resetting the KV cache, which preserves efficiency and context. When there's no textual input, the model outputs silence, enabling always-on, responsive speech generation. This modular design lets it pair with any LLM, maintaining high speech quality while delivering real-time, interruption-aware voice interactions for virtual assistants, robots, and real-time translation systems.
- Directly consumes LLM text-token streams, eliminating multi-stage ASR-to-TTS pipelines
- Control tokens enable user barge-in and mid-utterance interruptions without resetting the KV cache
- Produces silence when no text input exists, enabling always-on continuous speech synthesis
Why It Matters
VoiceChat-TTS could make voice assistants feel more natural by allowing real-time interruptions and continuous speech without latency tradeoffs.