VLATIM benchmark: VLMs plan like humans but fail at precise clicking in puzzle games

A new research paper from Dominik Helfenstein, Marco Menner, and Maximilian Triebel introduces VLATIM (Vision-Language Against The Incredible Machine), a benchmark designed to evaluate how well vision-language models (VLMs) perform human-like logical problem-solving in point-and-click puzzle games. The benchmark uses The Incredible Machine 2, a classic physics puzzle game that requires players to chain together objects like levers, ropes, and balls to achieve a goal. Unlike existing benchmarks that focus on discrete tasks, VLATIM targets the gap between high-level reasoning and continuous action spaces that require precise mouse interactions.

The benchmark is structured into five progressive parts, ranging from basic visual grounding (identifying objects and their properties) to full puzzle solving with multi-step manipulation. The results reveal a significant disparity: large proprietary models (like those from OpenAI and Google) demonstrated strong planning abilities—they could propose logical sequences of actions. However, they consistently failed at precise visual grounding, such as clicking on the exact pixel location of a small lever or correctly positioning an object. None of the tested models achieved human-level problem-solving, underlining that while VLMs can reason about physics, they cannot yet execute that reasoning in a fine-grained interactive environment. This work highlights a critical bottleneck for deploying VLMs in robotics, gaming, and any domain requiring precise physical interaction.