NVIDIA's PROBE trains VLMs to move objects and answer hidden-scene questions

Vision-language models (VLMs) ace static 2D grounding and spatial reasoning, but real-world questions often require uncovering occluded objects. NVIDIA's new paper formalizes this as Manipulation-Grounded Visual Question Answering (MG-VQA) and introduces PROBE, a complete benchmarking and finetuning framework for VLM agents that must grasp or push distractors before answering. To support this, they built PROBE-Sim, a high-fidelity tabletop simulator with everyday objects and a robotic arm equipped with grasping and pushing tools. From this they generated PROBE-Bench, an evaluation suite of 150 cluttered-scene tasks spanning 6 question types, where a VLM perceives, manipulates, and then answers.

The results show a consistent pattern across frontier VLMs: agentic tool-based methods (which can manipulate the scene) outperform perception-only baselines by an average of 8.0% on all task types. Building on this, the team designed PROBE-Agent, a finetuning recipe that distills successful trajectories from a powerful teacher foundation model into smaller open-weight models using a mixed data strategy that encourages manipulation-efficient question answering. These finetuned models beat their off-the-shelf agent baselines by 11.5% on average and show positive transfer to unseen objects and a held-out task. Crucially, the team validated sim-to-real transfer by deploying PROBE-Agent policies on real tabletop environments, demonstrating that physically interactive reasoning is not just a simulation curiosity but a practical stepping stone toward more capable home robots.