BayLing-Duplex lets LLMs listen and speak simultaneously with 92% success

BayLing-Duplex tackles a core limitation of current spoken chatbots: most are turn-based and rely on an external Voice Activity Detection (VAD) module to mark when the user is done speaking. This prevents natural overlapping speech, hesitation handling, and barge-in—hallmarks of human conversation. The team from (likely Chinese Academy of Sciences) designed BayLing-Duplex as a native full-duplex SpeechLM where a single autoregressive LLM internally decides when to listen, when to speak, and when to stop. The key innovation is adding just a few special tokens to the standard vocabulary, making the approach transferable across LLMs and reusable with existing training and serving stacks without architectural changes. Starting from the public GLM-4-Voice checkpoint and using only 400K full-duplex samples for fine-tuning followed by a lightweight DPO stage, the model learns to manage turn-taking end-to-end.

Results on the InstructS2S-Eval benchmark show BayLing-Duplex achieving 92% turn-taking success and 100% interruption success, while the speech-response quality score jumps from 2.17 (Moshi) to 3.39. Importantly, the model also matches or surpasses its turn-based counterpart on standard text benchmarks like Llama Questions, Web Questions, and Alpaca-Eval, proving that simultaneous listen-and-speak modeling does not degrade response quality. This work significantly advances the goal of truly natural, real-time speech interaction for AI assistants, removing the artificial pauses that plague current voice interfaces.