Dynamic Fusion-Aware Graph Convolutional Neural Network for Multimodal Emotion Recognition in Conversations
New AI model uses ODEs and a global vector to adaptively fuse text, audio, and visual cues for conversation analysis.
A research team including Tao Meng, Yuntao Shou, and five others has published a new paper on arXiv introducing the Dynamic Fusion-Aware Graph Convolutional Neural Network (DF-GCN). The model is designed for the complex task of Multimodal Emotion Recognition in Conversations (MERC), which involves analyzing emotions from multiple data streams like text, audio, and visual cues during dialogues. The core innovation addresses a key limitation in existing Graph Convolutional Network (GCN) approaches, which typically use fixed parameters to fuse these multimodal features. This static method forces a compromise across all emotion types, often limiting performance on specific categories.
To solve this, DF-GCN introduces two novel mechanisms. First, it integrates Ordinary Differential Equations (ODEs) into the GCN architecture. This allows the model to mathematically capture the dynamic, evolving nature of emotional dependencies and speaker interactions within a conversation network, rather than treating them as static. Second, the model employs a generated prompt from a Global Information Vector (GIV) of the conversation utterance. This GIV prompt actively guides the dynamic fusion process for multimodal features (text, audio, visual).
The combined result is a system that can adaptively change its internal parameters when processing each utterance. This means the network can effectively equip different parameters tailored for recognizing different emotion categories (e.g., joy, anger, sadness) during inference. This dynamic, context-aware fusion leads to more flexible and precise emotion classification. The researchers validated DF-GCN through comprehensive experiments on two public multimodal conversation datasets, where it demonstrated superior performance, confirming the significant benefit of its introduced dynamic fusion mechanism over previous static methods.
- Integrates Ordinary Differential Equations (ODEs) into Graph Convolutional Networks to model the dynamic evolution of emotional dependencies in conversations.
- Uses a Global Information Vector (GIV) to generate prompts that guide adaptive, context-aware fusion of text, audio, and visual features for each utterance.
- Achieves state-of-the-art performance on two public datasets by using dynamic parameters for different emotion categories, improving flexibility and generalization.
Why It Matters
Advances in MERC are critical for developing more nuanced and responsive AI for mental health apps, customer service bots, and human-computer interaction systems.