GameVerse: Can Vision-Language Models Learn from Video-based Reflection?

A research team led by Kuan Zhang has introduced GameVerse, a groundbreaking benchmark designed to test whether Vision-Language Models (VLMs) can learn from video-based reflection, much like humans do when playing games. Moving beyond traditional static evaluations, GameVerse implements a 'reflect-and-retry' paradigm where models watch their own failure videos and expert tutorials before attempting tasks again. The benchmark spans 15 globally popular games and features a dual action space that allows for both semantic commands (like 'jump') and direct GUI control, enabling comprehensive testing of visual reasoning and interaction capabilities.

In their experiments, the researchers found that VLMs showed significant performance improvements—up to 40% in some cases—when allowed to reflect on their failures and study expert demonstrations. The most effective approach combined watching failure trajectories with tutorial videos, creating what the paper describes as a 'training-free analogue to reinforcement learning plus supervised fine-tuning.' This suggests VLMs can internalize visual experience and refine their strategies without additional model training, potentially opening new pathways for developing more adaptive AI systems that learn from visual feedback loops in real-world scenarios.