Researchers launch VideoGAIA to stress-test AI video understanding

A team of researchers led by Fan Zhang from the University of Science and Technology of China has introduced VideoGAIA, a groundbreaking benchmark designed to push the limits of multimodal large language models (MLLMs) in video understanding. Published on arXiv, this benchmark addresses a critical gap in current evaluation methodologies, where models like GPT-5.5 and Kimi-K3 achieve over 90% accuracy on traditional benchmarks such as Video-MME. VideoGAIA shifts the paradigm by moving beyond single-turn video question answering to a more complex, agentic framework.

In this new paradigm, models must engage in multi-turn interactions, leveraging external tools to iteratively perceive videos, gather complementary information, and integrate multimodal evidence across turns. The benchmark comprises 271 tasks co-designed with humans, covering diverse and complex real-world scenarios. Each task is independently verified by three human experts to ensure correctness and appropriate difficulty. Despite the sophistication of leading models, none currently exceed 60% accuracy on VideoGAIA, highlighting a significant opportunity for improvement and innovation in next-generation MLLMs.