AI Struggles to Understand Human Emotions in New Sounds

Imagine an AI that claims it can tell if a song makes you happy or sad. Now, imagine playing it a version of that song you edited slightly—maybe you cut out the chorus. New research shows that AI often gets it wrong in these situations, even though it works perfectly fine on the original version you trained it on.

The study, published by researchers Jingyi Zhang and Xiaotong Yao, tested AI’s ability to predict emotions from audio in four tricky scenarios: new sounds (like switching from music to a baby crying), edited audio (like shortening a song), using a different sensor to measure your reaction (instead of your direct report), and predicting emotions for a person the AI hasn’t met before. In each case, the AI struggled to generalize—meaning it couldn’t reliably carry its skills over to new situations.

The researchers found that within the same type of audio, the AI could predict emotions pretty well—up to 84% as well as humans agreeing with each other. But when the sound changed even slightly, like using a completely different type of audio, the AI’s accuracy dropped by up to 80%. Even adding more data or using bigger AI models didn’t fully fix the problem. For example, when predicting emotions from physical reactions (like heart rate) instead of direct responses, the AI barely scratched the surface—only reaching a third of what’s possible.

The big takeaway? AI’s struggles to generalize aren’t always about needing more data or a bigger model. Sometimes, the problem is deeper—meaning the AI might never fully understand emotions in audio the way humans do. This matters because AI is increasingly used in apps, cars, and devices to react to how you feel, whether it’s recommending music based on your mood or alerting parents when a baby sounds upset.