Gemma, Qwen, MiniCPM share emotion neurons across speech and faces

A research team led by Xiutian Zhao, with Luqi Sun, Björn Schuller, and Berrak Sisman, published a paper on arXiv (2608.17102) investigating how modern multimodal foundation models (MFMs) process emotion across speech and faces. Using three open models—Google's Gemma-4-12B-it, OpenBMB's MiniCPM-o-4.5, and Alibaba's Qwen2.5-Omni-7B—they identified sparse decoder neurons, dubbed emotion-sensitive neurons (ESNs), that fire selectively for specific emotion categories. By probing with speech emotion recognition and facial expression recognition tasks, the team found both acoustic and visual ESNs distributed across the models' layers.

Their causal experiments showed these neurons matter: deactivating a visual ESN selectively impaired recognition of its associated facial emotion, while boosting its activation enhanced recognition of that emotion relative to others. Even more striking, acoustic and visual ESNs showed emotion-matched overlap and similar layer-wise distributions, and cross-modal interventions demonstrated bidirectional causal transfer—ESNs discovered in speech produced emotion-specific effects when applied to facial inputs, and vice versa. This provides one of the first activation-level analyses showing that speech and facial emotion recognition partially converge onto shared, sparse decoder-level components. The findings suggest emotion understanding in MFMs isn't fully siloed by modality; it emerges from common functional units that can be localized and manipulated without fine-tuning, opening new paths for interpretability and targeted control of emotion-related behavior in AI systems.