Spatially Grounded Long-Horizon Task Planning in the Wild
New benchmark reveals VLMs fail at 'where to act' planning, a critical bottleneck for real-world robots.
A team of researchers from institutions including KAIST and Microsoft has published a paper introducing a critical new benchmark for robot AI. The work, titled "Spatially Grounded Long-Horizon Task Planning in the Wild," addresses a fundamental flaw in how we evaluate Vision-Language Models (VLMs) like GPT-4V or Claude 3 for robotics. Current benchmarks let VLMs generate high-level action plans (e.g., "pick up the cup, then pour water") but fail to assess if these plans are spatially executable—they don't test if the AI can specify *where* in a cluttered scene the robot should actually interact.
To bridge this gap, the researchers created GroundedPlanBench. This novel benchmark jointly evaluates two capabilities: hierarchical sub-action planning (breaking down a task) and spatial action grounding (identifying precise interaction locations). Their evaluations reveal that spatially grounded long-horizon planning is a major bottleneck for current state-of-the-art VLMs, which struggle to connect abstract instructions to concrete, actionable spatial coordinates.
Alongside the benchmark, the team introduced Video-to-Spatially Grounded Planning (V2GP), an automated framework that leverages real-world robot video demonstrations to generate training data. This approach shows promise for significantly improving both the planning and spatial grounding performance of AI models. The research was validated not only on the new benchmark but also through real-world robot manipulation experiments, marking a concrete step toward robots that can autonomously plan and act in unstructured, "in the wild" environments.
- Introduces GroundedPlanBench, a novel benchmark evaluating both task decomposition and precise spatial grounding for robot actions.
- Reveals that current Vision-Language Models (VLMs) have a major bottleneck in spatially grounded long-horizon planning.
- Proposes V2GP, an automated data generation framework using robot videos to train models, validated in real-world experiments.
Why It Matters
This work is essential for moving AI planning from abstract language to actionable, real-world robot execution in homes and factories.