Beyond Pessimism: Offline Learning in KL-regularized Games

A team of researchers has published a breakthrough paper titled 'Beyond Pessimism: Offline Learning in KL-regularized Games' on arXiv. The work, led by Yuheng Zhang, Claire Chen, and Nan Jiang, tackles a core challenge in training AI agents: learning optimal behavior from a fixed, pre-collected dataset without interacting with the environment—a process known as offline learning. In competitive scenarios modeled as two-player zero-sum games, prior methods relied on 'pessimistic' value estimation to handle the 'distribution shift' problem, where the agent's learned policy deviates from the data-collection policy. This pessimism led to a slow statistical convergence rate of Õ(1/√n), meaning performance improved slowly as the dataset grew.

The new research introduces a 'pessimism-free' algorithm and analytical framework specifically for KL-regularized games, where policies are constrained to stay close to a reference policy. The key insight leverages the mathematical smoothness of KL-regularized best responses and a stability property of the Nash equilibrium. This allows their algorithm to achieve a dramatically faster Õ(1/n) sample complexity. In practical terms, this is a 10x improvement in data efficiency; to achieve a certain level of performance, the new method needs roughly the square root of the data required by old methods. Furthermore, the team proposes an efficient self-play policy optimization algorithm that, with a linear number of iterations, matches this optimal statistical rate, making it a practical tool for training more capable and sample-efficient AI agents for game theory and strategic reasoning tasks.