New AI Pinpoints Every Laugh, Sigh and Cough in a Recording

Human speech isn't just words. When you say "I'm fine" while sighing, the sigh is doing most of the talking. AI transcription tools are great at words, but they treat sounds like laughter, coughing or a sharp intake of breath as a fuzzy tag dropped somewhere near the sentence — with no sense of when it actually happened. That's a real loss, because timing is meaning: a laugh in the middle of a sentence says something different than a laugh right after it.

A research team led by Yuang Cao set out to fix that. They gathered 26 different categories of these non-word sounds from existing public datasets and built a large training set where every sound has a precise timestamp, using two AI models to double-check each other plus audio alignment and volume-based cleanup. They also created a human-verified test set called NVV-TimeBench: 667 short clips containing 1,094 separate events. The system itself predicts three things at once — where each word falls, what kind of sound it hears, and exactly when that sound begins and ends.

On that test set, the model scored about 71% accuracy in identifying and categorizing events, and it placed each one within roughly 60 milliseconds of the correct spot — faster than the blink of an eye. Notably, it beat several much larger, general-purpose audio AI models, and it held up reasonably well on a completely separate collection of recordings, suggesting it isn't just memorizing its training data.

So what's the point? Imagine meeting notes that capture not just what was said but the nervous laugh that followed a deadline question. Voice assistants that notice frustration and adjust their tone. Therapy or health apps that track emotional cues over weeks. More natural-sounding AI voices and better captions for deaf users. The catch: it's still a lab result, and roughly three in ten sounds were missed or mislabeled. There's no product yet — and an AI that reads emotion from your voice raises obvious privacy questions worth asking early.