AI That Talks and Listens at Once Just Got Better at Interrupting

Right now, most AI voice assistants work like a walkie-talkie: you talk, they wait, then they answer. Real conversation isn't like that. People interrupt, pause mid-sentence, murmur "mm-hmm" to show they're listening, and sometimes start talking before the other person is done. Teaching a machine to handle that gracefully is hard, mainly because there isn't enough recorded, labeled data showing who spoke when and why.

A research team led by Matthew Sun solved the data problem by manufacturing it. Their pipeline uses a large language model (AI that writes text) to script conversations event by event — deciding who speaks, what they say, and whether each moment is an interruption, a pause, or just a polite acknowledgment. Each line is turned into speech separately, then stitched onto a shared clock so the overlaps and silences are exactly measured. The system covers 42 conversational behaviors across English and Mandarin, including the tricky case of speech aimed at a third person in the room.

The payoff shows up in testing. After training on the generated conversations, the open-source full-duplex voice model Moshi — which can listen and speak simultaneously — correctly took its turn in 85% of cases, up from 44% before. Its frame-by-frame accuracy at knowing whether it held the conversational floor jumped from 46% to 88%. A separate detector that only hears current and past audio — no peeking ahead — identified when people started speaking or listening with roughly 80% accuracy.

The catch is that these are simulated conversations, not real recordings. Synthetic speech can miss the messy acoustics, accents, and emotional nuance of genuine human talk, so real-world performance may lag. The work is also a short paper headed to a technical conference, not a shipping product — meaning your phone's assistant won't gain this skill tomorrow. But it points clearly at a future where AI voice feels less like a command line and more like a chat.