New multimodal AI boosts audio sentiment analysis via multilingual transcripts and distillation

Automatically recognizing sentiment from speech remains challenging because it requires parsing both vocal tone and the meaning of words. A new paper by Durdun, Constantinescu, and Ionescu (accepted at KES 2026) tackles this by combining audio features with automatic transcriptions. Their approach uses an ASR tool to generate transcripts from speech, then translates them into multiple languages via machine translation. These multilingual text embeddings are fused with audio features using a cascaded cross-modal transformer architecture, integrating one modality at a time. This multimodal ‘teacher’ model achieves significant improvements in sentiment polarity classification on a large-scale dataset.

The authors also demonstrate knowledge distillation: they train a unimodal audio-only ‘student’ model to mimic the teacher’s outputs. This student outperforms a standard audio-only model while incurring no additional computational cost during inference—a crucial advantage for real-time or resource-constrained applications. Ablation studies confirm that both the original ASR transcripts and the translated versions contribute to the performance boost. The team has publicly released their code to enable reproducibility. This work highlights how readily available text generation tools can cheaply enhance audio understanding without sacrificing efficiency.