EB-CaP: Energy-based cache improves video emotion recognition at test time

Facial expression recognition (FER) in videos remains challenging because models must capture subtle, temporally evolving affective states that vary significantly between individuals. While vision-language models like CLIP offer transferable visual-semantic representations, they still degrade under subject-specific distribution shifts during inference. Existing test-time adaptation (TTA) approaches typically update model parameters on the fly, which increases latency and computational cost. Cache-based alternatives avoid parameter updates but need many target samples to form reliable class prototypes—a problem early in adaptation and for rare classes.

To address this, Masoumeh Sharafi and colleagues at ÉTS Montréal introduced EB-CaP (Energy-Based Cache Personalization), a subject-based online TTA method for video FER. EB-CaP leverages a lightweight energy-based model to sample class-specific prototypes directly from the current unlabeled target video, populating a personalized cache without storing diverse source prototypes or accumulating large amounts of target data. Its energy function uses only pretrained CLIP, computing similarities between target video embeddings and class text embeddings. The method also maintains positive and negative caches for reliable and uncertain embeddings, gated by an adaptive entropy gate and a diversity gate to prevent redundancy. Final predictions combine cache-derived scores with CLIP scores. Experiments on BioVid, StressID, and BAH benchmarks show EB-CaP outperforms state-of-the-art TTA methods while keeping computational and memory overhead low, making it practical for real-time video analysis.