Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition

A research team led by Zhexian Huang has published a novel AI architecture, named Atsuko, designed to solve core problems in multimodal emotion recognition. Current systems that fuse text, video, and audio often fail to capture genuine complementary signals, either relying too heavily on the best-performing single modality or using coarse fusion methods that lose fine-grained emotional detail. Atsuko addresses this by orthogonally decomposing each modality's features into distinct spectral bands—high, mid, and low-frequency components—creating a more nuanced representation for analysis.

Building on this decomposition, the model employs a dual-path routing mechanism that performs fine-grained, cross-band selection and cross-modal fusion. A key innovation is the Marginal Complementarity Module (MCM), which actively quantifies the performance loss if a specific modality were removed. This creates a 'complementarity distribution' that acts as a soft supervisor, dynamically guiding the model's attention toward modalities that provide unique, non-redundant information gains, thereby mitigating shortcut learning from dominant signals like text.

Extensive testing demonstrates Atsuko's effectiveness, achieving state-of-the-art or superior performance on five established benchmarks: CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMSv2, and MIntRec. This indicates a robust improvement in accurately interpreting complex emotional states from combined cues. The work, detailed in the arXiv preprint 2603.13340, represents a significant technical advance in making AI-based emotion recognition more sophisticated and reliable by fundamentally rethinking how different data streams should interact.