New proof shows target updates stabilize linear Q-learning convergence

Donghwan Lee, in a paper submitted to arXiv, offers a rigorous theoretical explanation for why periodic and soft target updates stabilize Q-learning with linear function approximation. Using exact switched linear system (SLS) dynamics induced by the Bellman maximum operator and the joint spectral radius (JSR) of switching matrix families, Lee proves that under explicit spectral and step-size conditions, these target-update mechanisms guarantee convergence to the exact projected Q-Bellman solution. This is significant because linear Q-learning is known to diverge in general, yet practitioners have long relied on target updates to stabilize training without a full theoretical understanding. The analysis is first carried out for deterministic linear Q-learning, where the target-update mechanism is most transparent, and then extended to stochastic reinforcement learning by replacing deterministic modes with sampled stochastic modes and adding noise analysis. The work provides a foundational certificate for the mean recursion, enabling future work on practical RL algorithms.

For practitioners, this means that popular techniques like periodic hard updates (e.g., in DQN) and soft updates (e.g., in actor-critic methods) now have a solid theoretical basis. The paper explicitly defines the conditions under which these updates ensure convergence, which could guide hyperparameter tuning and algorithm design. Lee's use of joint spectral radius is particularly elegant, as it captures the stability of the switching dynamics. While the paper focuses on linear Q-learning, it opens the door for extending these stability proofs to nonlinear function approximation. The stochastic extension shows that the core deterministic analysis carries over, making this a key reference for RL theorists and engineers alike. The work is available on arXiv and has been submitted to a leading conference.