Learning Physics from Pretrained Video Models: A Multimodal Continuous and Sequential World Interaction Models for Robotic Manipulation

A research team led by Zijian Song has published a groundbreaking paper titled 'Learning Physics from Pretrained Video Models,' introducing the PhysGen framework. This work tackles a core challenge in robotics—the scarcity of large-scale training data—by ingeniously repurposing existing foundation models. Instead of training robots from scratch, PhysGen uses pretrained autoregressive video generation models as a proxy for a physics simulator. The key innovation is a 'multimodal continuous representation' that bridges the gap between discrete video frames and continuous robotic control by unifying video and action data into shared physical tokens. This allows the system to transfer implicit physical knowledge, such as object dynamics and permanence, learned from vast video datasets directly to manipulation tasks.

Technically, PhysGen incorporates several methods for efficient convergence, including causal masking, inverse kinematics, Lookahead Multi-Token Prediction (L-MTP), and key-value caching. Its performance is validated on major benchmarks: it surpassed robust baselines OpenVLA and WorldVLA by 13.8% and 8.8%, respectively, on the Libero and ManiSkill benchmarks. Most notably, in real-world tests, PhysGen matched the performance of large-scale action-pretrained models like π₀ without requiring any prior action-specific pretraining, excelling in physically complex tasks like grasping transparent objects. This demonstrates a scalable, data-efficient path toward more generalizable robotic manipulation by extracting 'physical intuition' from the world's video data.