Three-stage training gives LLM agents internal world models for planning

A team of researchers led by Xuan Zhang from multiple institutions has published a paper proposing a new training paradigm that gives large language model (LLM) agents genuine internal world models for planning. Standard agents reactively choose actions without simulating future consequences, unlike humans who perform 'what-if' reasoning. The authors identify a 'format-capability gap' where simply fine-tuning on look-ahead traces leads to superficial mimicry. Their solution is a three-stage training pipeline: World Model Agentic Mid-Training (WM-AMT) injects latent predictive capabilities into the policy; Format-Eliciting SFT (FE-SFT) structures that capability into explicit verbalized rollouts; and Foresight-Conditioned Reinforcement Learning (FC-RL) refines calibration and utility of the simulated futures. The paradigm trains a single autoregressive model to produce both a verbalized prospective state rollout and a plan-conditioned success estimate, effectively a textual analogue of the Q-value from reinforcement learning.

Evaluated on search and mathematical reasoning tasks, the approach consistently outperforms other training baselines. The results demonstrate that effective internal world modeling in LLM agents requires a capability-first training pipeline to achieve grounded and calibrated foresight. This work addresses a critical limitation in current LLM agents: their inability to engage in deliberate planning before committing to actions. By internalizing future simulation, the paradigm could significantly improve performance on long-horizon tasks in robotics, automated reasoning, and interactive environments. The paper is available on arXiv (2606.27483) as a preprint, pending review.