Policy-Guided World Model Planning for Language-Conditioned Visual Navigation

Researchers Amirhosein Chahe and Lifeng Zhou have introduced PiJEPA, a two-stage AI framework designed to solve a core challenge in robotics: getting a robot to navigate to a visually specified goal using only natural language instructions. Existing methods typically rely on either reactive policies, which fail at long-term planning, or world models, which struggle to initialize actions in complex environments. PiJEMA elegantly combines both approaches. Its first stage involves fine-tuning a generalist navigation policy called Octo, augmented with a frozen vision encoder like DINOv2 or V-JEPA-2, on the CAST dataset. This creates a smart 'policy prior' that suggests probable actions based on the current camera view and the user's command.

In the second stage, this informed action distribution is used to 'warm-start' a sophisticated planner. Instead of searching randomly, the planner uses the policy's suggestions to initialize a Model Predictive Path Integral (MPPI) algorithm, which then plans over a separately trained JEPA world model. This world model predicts future states in the vision encoder's latent space. By starting the search from a smart guess rather than a random one, PiJEPA's planner converges much faster to high-quality action sequences that successfully reach the goal. Experiments demonstrate that this hybrid approach significantly outperforms using either the policy or the world model alone, leading to more accurate and reliable robot navigation that faithfully follows complex instructions.