New 65-page survey maps the math behind modern RL algorithms

A team of eight mathematicians and ML researchers—Denis Belomestny, Alexander Gasnikov, Egor Gladin, Alexey Naumov, Artemy Rubtsov, Yuri Sapronov, Daniil Tiapkin, and Nikita Yudin—has released a 65-page survey on arXiv that systematically organizes the mathematical structures underpinning modern reinforcement learning (RL). The paper, titled "Mathematical methods of reinforcement learning," starts with Markov decision processes (MDPs) and the Bellman operators, emphasizing contraction mappings, monotonicity, and fixed-point theory to explain convergence guarantees and rates for value iteration, policy iteration, and temporal-difference methods. It then moves to an optimization perspective, covering stochastic approximation, martingale methods, convex duality, and the role of regularization in mirror/proximal algorithms.

The survey also tackles function approximation in both linear and nonlinear settings, including stabilization techniques, error decomposition, and sample-complexity analysis using concentration inequalities for dependent data and mixing processes. Off-policy evaluation and learning are treated alongside constrained RL and constrained MDPs (CMDPs). Throughout, the authors unify algorithmic templates under common operator and variational lenses, providing both finite-sample bounds and asymptotic results. The paper is intended as a mathematical entry point for researchers in probability, optimization, and statistics who are interested in RL—offering a rare level of rigor and breadth in one document.