What-If World benchmark exposes fatal flaw in AI video world models

A team of researchers led by Kunlin Cai has released What-If World, a benchmark that rigorously tests whether video generation models understand causality. The core idea: given two prompts describing the same scene with exactly one physical detail changed (e.g., an object is red vs. blue), the model's output videos should diverge in a physically consistent way. Existing benchmarks score videos individually and miss this failure mode, so the authors created 319 carefully designed prompt pairs drawn from the nuScenes (driving) and DROID (robotic manipulation) datasets. Each pair is scored using APEO, a four-part rubric that checks: Adherence (video matches its prompt), Physics (internal consistency), Environment (shared background preserved), and Outcome (the correct difference emerges).

Results across nine state-of-the-art models are sobering. No system exceeded 52% on the paired score, and open-source models clustered near 28%. Every tested model failed on a large fraction of causal interventions, indicating substantial room before these models can reliably support action-conditioned simulation or model-based planning. Notably, performance appeared to track the visual prominence of the intervention rather than its physical complexity: visually subtle changes scored as low as 14.2%, while obvious ones reached 40.4%. This suggests current models rely on superficial appearance cues rather than genuine physical reasoning. The findings challenge the notion that video generation models can serve as trustworthy world simulators for embodied AI tasks.