ModeratorLM improves multi-party voice agent turn-taking by 40%
New speech LLM cuts false interruptions by 70% in group conversations
Turn-taking in multi-party spoken interactions remains a major hurdle for voice-based AI, especially when multiple participants compete for the floor. Researchers introduce ModeratorLM, a voice agent that conditions its turn-taking behavior on an explicitly assigned role. The system is built on a speech large language model that operates in a chunk-wise streaming manner for real-time processing. A reasoning-augmented variant incorporates chain-of-thought reasoning over conversational context and the assigned role, allowing the agent to decide when to speak, listen, or wait.
The team created RolePlayConv, a large-scale synthetic dataset of spoken multi-party conversations with diverse assistant roles. Experiments on real-world meeting data and RolePlayConv demonstrate that ModeratorLM improves turn-taking precision by over 40% and recall by more than 70%, while substantially reducing false-positive interruptions compared to non-role-conditioned baselines. The work, accepted at Interspeech 2026, paves the way for more natural group interactions with voice assistants in meetings, call centers, and social robots.
- Uses a speech LLM with chunk-wise streaming for real-time multi-party turn-taking.
- Improves turn-taking precision by over 40% and recall by over 70% vs. baselines.
- Introduces RolePlayConv, a large synthetic dataset of multi-party conversations.
Why It Matters
Enables voice agents to handle group conversations naturally, reducing awkward interruptions in meetings and calls.