Privacy-safe group emotion recognition uses audio-video, not individual cues

Traditional emotion recognition relies on individual-level data like facial expressions, eye gaze, or voice patterns — raising serious surveillance and privacy concerns. This thesis, led by Anderson Augusma, flips the approach by focusing exclusively on group-level emotions. Instead of identifying how each person feels, the system reads collective audio-visual signals from a scene to infer the overall emotional state (e.g., “excitement” or “tension”) without singling out anyone. Two complementary frameworks are proposed. The first uses cross-attention to fuse audio and video features, then applies Frames Attention Pooling (FAP) to aggregate over time. Synthetic data augmentation and ablation studies confirm robustness in real-world, in-the-wild conditions.

The second framework, Variational Encoder Multi-Decoder (VE-MD), learns a shared latent space for both emotion classification and structural representation prediction (like body and face cues). Two decoding strategies are explored: a DETR-based transformer decoder and a heatmap-based approach. By comparing these, the thesis investigates how much structural information is needed for group vs. individual emotion recognition — and shows that competitive performance is achievable without using any individual-level features as input. The three main contributions are clarifying the role of multimodality and structural cues in group-level affective computing, introducing two novel architectures, and demonstrating that privacy-safe GER is feasible without sacrificing accuracy. This work opens the door to ethical deployment of emotion AI in public spaces like stadiums, airports, or classrooms.