New RL Theory Proves Quantile Methods Hit Optimal Statistical Efficiency

A new paper from researchers Zijie Cheng, Yang Peng, and Zhihua Zhang tackles the statistical foundations of quantile-based distributional reinforcement learning. The authors study the quantile-projected distributional Bellman equation, deriving a fixed-point representation of the return distribution. Using a generative model and an empirical Markov decision process, they construct an estimator that achieves a non-asymptotic error bound scaling as O(√(m/n)) under the supremum W∞ metric, where m is the number of quantiles and n is the number of samples. This implies the quantile-based policy evaluation problem is sample-efficient, attaining the optimal parametric √n convergence rate—the fastest possible for parametric problems. The work further characterizes the asymptotic distribution of quantile parameters, showing it matches the semiparametric efficiency bound, meaning no estimator can achieve lower asymptotic variance.

Beyond the fixed-dimensional setting, the authors investigate the regime where the number of quantiles m grows with sample size. They prove the limit covariance structure matches the semiparametric efficiency bound of the nonparametric model, establishing that quantile estimators remain asymptotically efficient even in the infinite-dimensional limit. Finally, a Berry-Esseen theorem is proved for smooth functionals of the quantile-projected return distribution, providing a foundation for statistically valid inference. For RL practitioners, this means quantile-based methods like those used in IQN or QRDQN are not just practical heuristics—they are theoretically optimal tools for understanding the full distribution of returns, opening the door to more reliable uncertainty quantification and risk-sensitive decision-making.