Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition

A team of researchers has introduced Crab (Contrastive Representation and Multimodal Aligned Bottleneck), a new AI architecture designed to solve a persistent problem in Speech Emotion Recognition (SER). While most current models apply supervision only at the final classification layer, Crab uses a novel Multi Layer Contrastive Supervision (MLCS) strategy. This technique injects contrastive learning signals at multiple network layers, forcing the model to learn emotionally discriminative features throughout its entire depth without adding parameters at inference time. The model is bimodal, fusing powerful speech representations from WavLM with textual context from RoBERTa within a Cross-Modal Transformer framework.

Crab was rigorously tested on three benchmark datasets representing a spectrum of emotional naturalness: IEMOCAP (acted), MELD (TV dialogues), and MSP-Podcast 2.0 (highly natural). The results showed that Crab consistently outperformed strong unimodal and multimodal baseline models across all tests. Its performance gains were most pronounced under the challenging conditions of natural, spontaneous speech and severe class imbalance—a common real-world scenario where data for emotions like 'sadness' far outweighs 'joy'. To handle this imbalance, the team also employed a weighted cross-entropy loss during training. The findings validate MLCS as a robust, general-purpose strategy for building more reliable SER systems that work outside the lab.