Acoustic and Semantic Modeling of Emotion in Spoken Language
New research combines acoustic and semantic data for better emotion recognition and synthesis in AI speech.
A new PhD thesis by Soumya Dutta tackles a core challenge for next-generation AI: emotional intelligence in spoken language. The research, titled 'Acoustic and Semantic Modeling of Emotion in Spoken Language,' presents a multi-pronged approach to help AI systems better understand and generate the emotional cues in human speech. It moves beyond simple sentiment analysis by jointly modeling both the acoustic properties (tone, pitch) and semantic meaning (words, context) of speech, which is critical for nuanced emotion recognition.
The thesis introduces several key technical contributions. First, it proposes novel pre-training strategies that incorporate acoustic and semantic supervision to learn speech representations rich with affective cues. A significant breakthrough is a speech-driven framework that enables large-scale emotion-aware text modeling without relying on costly, manually annotated datasets. For conversational AI, Dutta developed hierarchical architectures using cross-modal attention and mixture-of-experts fusion to integrate emotional context across dialogue turns.
Finally, the work presents a practical framework for controllable speech-to-speech emotion style transfer. This 'textless and non-parallel' system can transform the emotional tone of a speech sample—making a neutral statement sound happy or sad—while meticulously preserving the original speaker's identity and the linguistic content. The results show this synthesized emotional speech can also be used to augment training data, further improving the performance of emotion recognition models.
- Proposes a pre-training framework for emotion-aware text modeling without manually labeled corpora, enabling scalable training.
- Develops hierarchical architectures with cross-modal attention for recognizing emotion in multi-turn conversations.
- Introduces a textless speech-to-speech emotion transfer system that changes emotional tone while preserving speaker identity and words.
Why It Matters
This research is foundational for creating more natural, empathetic, and effective voice assistants, customer service bots, and therapeutic AI tools.