Study finds EmoPrefer benchmark rewards style over substance in emotion AI evaluation

Researchers from multiple institutions conducted a systematic shortcut audit of the EmoPrefer benchmark, part of the MER2026 MER-Prefer track, which evaluates multimodal emotion understanding by asking models to predict which AI-generated emotion description a human would prefer. Using content-blind probes, they found that a simple logistic regression model using only description length and generator identity—without processing any text, video, or audio—achieved a Weighted Accuracy Factor (WAF) of 65.8, compared to 66.8 for LoRA-finetuned 7B text and audio-visual judges. The generator identity was recoverable from description text with 99.5% accuracy, and every candidate pair contrasted two distinct generators. Human preference labels agreed with a simple per-generator win-rate prior on 66% of the evaluated pairs. When the human label conflicted with this prior, trained judges still followed the style prior on 63-80% of pairs.

On a length-matched subset that neutralized verbosity bias, the tested media configurations yielded no statistically significant improvement over the blind probe. An ODIN-inspired diagnostic that decoupled the style shortcut left its content head near chance. These results demonstrate that current EmoPrefer scores can be reached without verifying any description against the video content—the benchmark can be gamed by exploiting style cues like description length and generator identity. The authors recommend source-balanced pairing, strict length control, counter-stereotypical sliced reporting, and multi-annotator consensus for future cross-generator evaluations. Code for the audit is publicly available.