What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty

Aran Nayebi's groundbreaking paper, 'What Capable Agents Must Know: Selection Theorems for Robust Decision-Making under Uncertainty,' provides the first formal proof that capable AI agents must develop internal world models to perform well. Published on arXiv, the work addresses a fundamental question in AI theory: what internal structure is necessary for competent action under uncertainty? While classical results showed world models could implement optimal control, Nayebi's selection theorems prove they are actually required for agents to achieve low average-case regret across structured prediction tasks, covering stochastic policies and partial observability without assuming optimality.

The technical core reduces predictive modeling to binary betting decisions, demonstrating that regret bounds limit probability mass on suboptimal bets, thereby enforcing the predictive distinctions needed to separate outcomes. In fully observed settings, this yields approximate recovery of the interventional transition kernel; under partial observability, it implies necessity of belief-like memory and predictive state. This work directly addresses an open question in prior world-model recovery research and provides mathematical justification for the architecture of modern AI agents (systems that can take actions), potentially guiding development of more robust and interpretable systems like those built on frameworks such as Llama 3 or Claude 3.5.