C²MOE mixture-of-experts beats SOTA on incomplete multimodal emotion recognition

Multimodal Emotion Recognition in Conversations (MERC) typically assumes all modalities—audio, text, and video—are fully available. But real-world data often loses channels due to transmission errors or user behavior, which severely degrades performance. Existing methods try to fix this with cross-modal consistency learning but ignore modality complementarity, leading to biased reconstructions. To solve this, the researchers introduce C²MOE, a novel framework that unifies representation learning and missing modality imputation within a principled information-theoretic framework.

C²MOE factorizes multimodal knowledge into consistency and complementarity components via interaction-aware experts. Consistency is captured by maximizing cross-modal predictability, while complementarity is preserved by maximizing conditional entropy between modalities. The framework then uses a dual-branch prediction mechanism: one branch aligns imputed features with the joint distribution by minimizing uncertainty, and the other exploits modality-unique cues via entropy maximization. A learnable reweighting module dynamically assigns importance scores to each expert's output for adaptive fusion. Extensive experiments on multiple MERC benchmarks demonstrate that C²MOE consistently outperforms state-of-the-art methods across various missing-modality settings, validating its robustness and generalization.