CogAudio-LLM gives AI ears that truly understand emotions

Audio Language Models (ALMs) have mastered semantic understanding but still fail at nuanced emotional interactions—text dominates, and acoustic subtleties like sarcasm or distress are lost. To fix this, Zhixian Zhao and colleagues at Northwestern Polytechnical University introduce CogAudio-LLM, a framework that explicitly separates what is said from how it is said. Their LIME-440K dataset contains 440,000 utterances where the same sentence is spoken in multiple emotions (e.g., happy, sad, angry), enabling the model to learn acoustic-semantic decoupling without lexical bias. This dataset is a first-of-its-kind benchmark for emotional audio reasoning.

CogAudio-LLM’s core is the Emotion-Inference-Perception-Synthesis (EIPS) pipeline, a 4-step Chain-of-Thought process borrowed from cognitive psychology. It first infers the speaker’s emotional state, then perceives acoustic patterns, synthesizes a logical rationale, and finally generates an empathetic response. To avoid the computational cost of explicit reasoning at inference, multi-stage training first teaches EIPS via supervised fine-tuning, then distills this logic into an implicit generation. The final innovation, DR-SAPO (Dual-Route Soft Adaptive Policy Optimization), dynamically weights the trade-off between strict logical reasoning and natural, emotionally resonant language. Experiments show CogAudio-LLM outperforms GPT-4o and other ALMs on both emotion recognition accuracy and user-rated empathy, setting a new bar for affective audio AI.