Grounding Social Perception in Intuitive Physics

A team from MIT and Harvard, including Joshua Tenenbaum, has published a paper proposing that human social perception is fundamentally grounded in intuitive physics. To test this, they created the PHASE (PHysically grounded Abstract Social Events) dataset—a large collection of procedurally generated 2D animations showing simulated agents interacting in environments with varied geometry, goals, and relationships (friendly, adversarial, neutral). This systematic variation allows for rigorous testing of how both humans and AI infer social information from physical actions.

They then introduced SIMPLE, a computational model that integrates Bayesian inverse planning with physics simulation. Unlike standard vision-language models (VLMs) that perform pattern matching, SIMPLE reasons by inverting a generative model: it simulates possible agent goals and physical constraints to explain observed trajectories. In experiments, SIMPLE's inferences closely matched human judgments across diverse scenarios. In contrast, strong feedforward baseline models, including modern VLMs, and physics-agnostic inverse planning models failed to achieve human-level performance or alignment.

These results challenge the dominant paradigm in AI social reasoning. The success of SIMPLE suggests that to truly understand social scenes—like inferring an agent's goal or relationship from their movement—an AI must reason about the physical plausibility of actions and the causal structure of the environment. This points toward a future where more robust, human-like AI social cognition is built not on larger datasets alone, but on integrated models of physics, planning, and psychology.