New distributional TD learning theory gives bootstrap inference Gaussian guarantees

Reinforcement learning often needs to estimate the entire distribution of future returns, not just the expected value—especially for risk-sensitive applications like finance, healthcare, and autonomous driving. Distributional temporal-difference (TD) learning does exactly this, but until now, no one had proven a valid statistical inference framework for the estimated distribution. Yang Peng and Liangyu Zhang's new paper, 'Online Inference in Distributional Temporal-Difference Learning,' fills that gap with a rigorous theoretical analysis of the nonparametric estimator under a fixed policy.

Working in the Cramér space—a natural space of probability distributions with finite first moment—the authors prove that the root-T error of the Polyak-Ruppert averaged estimator converges weakly to a centered Gaussian random element. More importantly, they show the bootstrap, resampled conditionally on the observed trajectory, converges to the same Gaussian limit. This validates bootstrap confidence intervals for smooth functionals such as variance, CVaR, expected shortfall, and expectiles. For nonsmooth functionals like quantiles, the authors develop a local asymptotic theory over T^-1/2 neighborhoods of thresholds, extending inference to quantile-based risk measures. With 61 pages of proofs, this paper lays the statistical foundation for making distributional RL trustworthy in practice.