Online Statistical Inference of Constant Sample-averaged Q-Learning

A team of researchers has introduced a novel framework for performing statistical inference on a modified reinforcement learning algorithm called Constant Sample-averaged Q-learning. The core innovation is adapting the functional central limit theorem (FCLT) to enable the construction of confidence intervals for the algorithm's Q-values—the estimates of future rewards that guide an AI agent's decisions. This addresses a critical weakness in traditional Q-learning: its performance can be highly unstable and suffer from high variance, especially in environments with noise or sparse rewards. By using a technique called random scaling, the method quantifies the uncertainty of these learned values in real-time.

The researchers validated their framework by comparing it to traditional Q-learning on two benchmark problems. The first was a simple grid world, serving as a proof-of-concept. The second was a more complex dynamic resource-matching problem, representing a real-world application. The experiments reported coverage rates and confidence interval widths, demonstrating that their modified approach provides statistically reliable uncertainty estimates. This work, presented at the Reinforcement Learning Safety Workshop (RLSW) at RLC 2024, provides a formal statistical tool to assess the reliability of an AI agent's learned policy before deployment in safety-critical scenarios.