Frozen multimodal embeddings predict personality traits from video interviews with 19% lower error
Researchers use CLIP, Whisper, and RoBERTa to assess HEXACO traits from 30-second video clips.
Kuo-En Hung, Hung-Yue Suen, Shih-Ching Yeh, and Hsiang-Wang Wang present a solution for the ACM Multimedia AVI Challenge 2026, using frozen multimodal encoders (CLIP for visual features, Whisper for acoustic features and transcripts, and RoBERTa, E5, and DeBERTaV3 for text) to predict HEXACO personality traits and cognitive ability from asynchronous video interviews. Their trait-specific late-fusion model achieved an average validation MSE of 0.2696, a 19.1% improvement over the official baseline of 0.3334. For cognitive ability, the authors' multimodal ensemble reached 0.5313 accuracy, while a compact subject-attribute baseline reached 0.5781—both above the official baseline of 0.4062. The paper interprets the cognitive results as evidence of possible subject-attribute shortcuts in the validation split rather than robust inference from AVI content.
- Achieved 19.1% MSE reduction for HEXACO personality traits using frozen multimodal embeddings (CLIP, Whisper, RoBERTa) with trait-specific late fusion
- Cognitive ability classification reached 0.5781 accuracy but may rely on dataset shortcuts rather than genuine cognitive inference
- Frozen encoder approach avoids fine-tuning large models on small labeled datasets, enabling efficient representation learning for psychological assessment
Why It Matters
Enables scalable, non-intrusive psychological assessment from video interviews, but highlights bias risks in automated hiring tools.