New paper: VLA robots fail to prove they understand physics

A new position paper from researchers at the University of Sydney (Taozhao Chen, Ian Manchester, and Huaming Chen) takes a critical look at Vision-Language-Action (VLA) models—the AI systems that combine vision and language understanding to control robots. These models, built on pretrained vision-language models (VLMs) like GPT-4V or CLIP, have shown rapid performance improvements on standard robot manipulation benchmarks. Many in the field interpret these gains as evidence that semantic knowledge learned from internet-scale data transfers to physical world understanding.

The authors challenge this assumption. By decomposing VLA policies into two components—semantic mapping (recognizing objects and commands) and physical action decision (choosing forces, trajectories, etc.)—they show that the dominant evaluation metric, task success rate, cannot distinguish between these capabilities. Improvements could come from semantic pattern matching, distributional overlap in training data, or genuine physical generalization, but current protocols can't tell which. The paper also identifies 'narrative drift,' where each new model inherits and amplifies earlier interpretations without causal verification. To fix this, the authors propose evaluation designs with controlled variation to separately measure semantic and physical generalization, aiming to clarify whether VLMs act as useful semantic interfaces or true sources of physical competence.