New Test Shows AI Voice Assistants Still Struggle With Real Chinese Conversations

AI voice assistants have gotten much better at understanding what people say. But most tests only check if the assistant gives the "right" answer in a clean, quiet setting. Real conversations are messier. People interrupt, background noise happens, and tone of voice changes meaning. A new research paper introduces TELEVAL, a benchmark designed to test AI assistants in realistic Chinese spoken conversations that aren't scripted and rely on audio cues, not just words.

The results aren't great for AI. While the tested models did fine on simple semantic tasks—like answering a fact-based question—they struggled when the audio was noisy or when the situation required an appropriate social response. For example, if someone sighs or sounds hesitant, a human would adjust their answer. The AI often ignored that context and gave a robotic reply. In fact, the researchers found a recurring problem they call the "Caption Trap": the AI would simply describe the sound it heard instead of responding conversationally, like a narrator rather than a participant.

Why does this matter? Because voice assistants are moving beyond smart speakers and into phone calls, customer service, and even therapy-like conversations. If a system can't handle the natural flow of speech—with its pauses, emotions, and ambient noise—it will frustrate users and limit what we can trust it to do. The researchers argue current models are "insufficiently aligned" with natural interaction, meaning they still need significant work before they can truly talk with us, not just respond to commands.

The good news is TELEVAL provides a way to measure these shortcomings. That's a step toward building better assistants that don't just understand words, but understand people. For now, it's a reminder that the smooth voice assistants in ads are still far from the messy reality of everyday conversation.