CF-GRPO boosts video AI reasoning without human temporal labels
New method uses consensus frames to tell video AI what visual evidence to trust.
Video multimodal large language models (Video-MLLMs) struggle to determine which visual evidence supports their answers. Existing reinforcement learning approaches use outcome-only rewards, offering little guidance on frame-level reasoning. A new paper from researchers including Chengwen Liu, Zhe Huang, and Tat-Seng Chua introduces CF-GRPO (Consensus Frame GRPO), inspired by multisensory integration in biology—consistent cues enhance salience and reliability.
CF-GRPO constructs a consensus frame prior from three intrinsic video signals: temporal coverage, scene-transition cues, and query-conditioned visual relevance. It then computes a model-side frame-use score from visual and response representations, optimizing agreement through the Consensus Frame Reward (CFR). With salience-aware sparse aggregation and distribution sharpening, CFR provides high-contrast reward signals without requiring any human temporal annotations. Experiments show VideoCFR achieves competitive performance across complex video reasoning benchmarks, improving multiple metrics over representative Video-MLLM and RL baselines. The consensus prior also gives an interpretable view of which evidence frames the model emphasizes during training.
- CF-GRPO introduces a process-level reward that guides video LLMs to focus on the most relevant frames without human annotation.
- The consensus frame prior combines temporal coverage, scene transitions, and query relevance—no extra labeling needed.
- VideoCFR outperforms existing Video-MLLM and RL baselines on complex video reasoning benchmarks.
Why It Matters
Makes video AI reasoning more transparent and data-efficient by automatically identifying key visual evidence.