New statistical inference method boosts distributional RL efficiency
Researchers propose online inference for quantile temporal difference learning, cutting memory costs by 80% in distributional RL.
A new arXiv paper introduces online inference methods for quantile temporal difference learning (QTD) in distributional reinforcement learning. The authors establish functional central limit theorems for both synchronous and asynchronous QTD, showing that averaged iterates converge weakly to a rescaled Brownian motion. Their random-scaling-based inference procedure builds an asymptotically pivotal statistic using information along the entire QTD path—and it can be computed online without storing the full trajectory, substantially reducing memory requirements and enabling efficient statistical inference.
- Proposes online inference method for quantile temporal difference learning (QTD) in distributional RL
- Reduces memory requirements by 80% by computing statistics without storing entire trajectories
- Uses rescaled Brownian motion and random scaling to enable real-time statistical analysis
Why It Matters
Enables efficient real-time statistical analysis of reinforcement learning models with 80% less memory usage