Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection

A team of researchers from Tel Aviv University and Meta AI has published a groundbreaking study exposing a fundamental flaw in today's most advanced video-understanding AI models. The paper, "Action Without Interaction," introduces a novel, large-scale dataset of over 20,000 annotated videos from the Something-Something-V2 dataset, where 250 human annotators meticulously labeled the precise moments ('contact' and 'release') when a hand touches or lets go of an object. When state-of-the-art models from OpenAI (GPT-4), Google (Gemini), and Alibaba (Qwen) were tested on this dataset, they demonstrated a stark disconnect between semantic understanding and physical reasoning.

While the models could reliably describe the actions and name the objects in the videos—a form of intuitive, System 1 pattern recognition—they consistently failed at the core task. They could not accurately identify the specific video frame where the physical interaction began or ended, nor could they correctly localize the event within the scene. The researchers term this 'shortcut learning,' where success in high-level description masks a failure in low-level physical grounding. This suggests these LMMs lack the System 2 cognitive foundations needed to reason about basic physical primitives, a capability essential for true understanding of dynamic, real-world scenes.