Watch Before You Answer: Learning from Visually Grounded Post-Training

A team of researchers has published a paper titled 'Watch Before You Answer: Learning from Visually Grounded Post-Training,' exposing a critical flaw in how AI models are trained to understand videos. The study found that 40-60% of questions in widely used long-video understanding benchmarks, like those used to evaluate models such as GPT-4V or Claude 3, do not actually require watching the video; they can be answered using text cues alone. This 'linguistic bias' is also pervasive in the datasets used for post-training, which is the final fine-tuning stage for models like Llama 3. This means progress in video AI has been overstated, as models were learning to exploit textual shortcuts rather than developing true visual comprehension.

The researchers' solution is a simple but effective data curation technique called VidGround. Instead of using entire, potentially biased post-training datasets, VidGround filters for only the questions that demonstrably require visual information to answer. When applied alongside standard Reinforcement Learning (RL) post-training algorithms, this method boosted performance by up to 6.2 percentage points on video tasks, while using only 69.1% of the original data. Crucially, this simple data-focused approach outperformed several more complex post-training techniques, highlighting that the primary bottleneck for better video AI is not algorithmic complexity but data quality. The findings underscore that for Vision-Language Models to advance, both evaluation benchmarks and training data must be rigorously grounded in visual reality.