MTR-DuplexBench: Towards a Comprehensive Evaluation of Multi-Round Conversations for Full-Duplex Speech Language Models
New benchmark exposes major flaws in models like GPT-4o and Gemini Live, showing they struggle with multi-turn, overlapping speech.
A research team from The Chinese University of Hong Kong, Shenzhen, and Sun Yat-sen University has published MTR-DuplexBench, a comprehensive new benchmark designed to evaluate Full-Duplex Speech Language Models (FD-SLMs). These models, like OpenAI's GPT-4o and Google's Gemini Live, enable real-time, overlapping conversations where users can interrupt the AI, mimicking natural human dialogue. The benchmark addresses a critical gap: existing tests focus on single-turn interactions, failing to capture the complexities of multi-round communication where context and turn boundaries become blurred.
MTR-DuplexBench's methodology is its key innovation. It doesn't just analyze raw audio streams; it segments continuous full-duplex dialogues into discrete conversational turns for granular, turn-by-turn assessment. The evaluation spans four critical dimensions: conversational features (like latency and interruption handling), dialogue quality, instruction following, and safety. The team's experiments, accepted at ACL 2026, reveal that even state-of-the-art FD-SLMs exhibit significant performance degradation across multiple conversation rounds and struggle to maintain consistency across these different evaluation aspects.
The findings underscore that building a truly natural conversational AI is more challenging than optimizing for single responses. This benchmark provides the first standardized tool for developers to diagnose weaknesses in real-time interaction, moving beyond simple transcription accuracy or single-turn chat quality. By making the code and data publicly available, the researchers aim to accelerate progress toward more robust, context-aware, and reliable voice-based AI assistants.
- Benchmark segments continuous, overlapping speech into discrete turns for turn-by-turn analysis of models like GPT-4o.
- Evaluates four key dimensions: conversational features, dialogue quality, instruction following, and model safety.
- Initial results show current FD-SLMs suffer from performance inconsistency across multiple conversation rounds.
Why It Matters
Provides the first standardized test for real-time, interruptible AI conversations, guiding development of more natural voice assistants.