Bias-corrected stochastic approximation achieves 1/2 exponent convergence rate

A new paper from researchers Dhruv Sarkar and Vaneet Aggarwal tackles one of the fundamental bottlenecks in stochastic approximation: the slow convergence of two-time-scale algorithms under non-expansive maps. These algorithms are widely used in reinforcement learning, online optimization, and adaptive control, where a fast inner loop tracks a time-varying equilibrium while a slow outer loop updates the target. Until now, practitioners have observed a frustrating k^{-1/4+o(1)} last-iterate mean-square residual, which limits sample efficiency.

The authors first prove that this 1/4 exponent is actually a worst-case lower bound for any fixed stepsize schedule, ruling out simple tweaks. They then introduce a theoretical preconditioning technique that corrects the bias in the slow oracle by converting the first-order dependence on fast tracking error into second-order. In a nested Tikhonov-Krasnoselskii-Mann algorithm, this bias correction lifts the total-sample rate from T^{-1/4+o(1)} to T^{-1/3+o(1)} — a meaningful improvement for large-scale problems.

Perhaps more exciting for practitioners, the authors show that the inner-loop cost can be eliminated entirely in smooth derivative-oracle models. Their single-loop algorithm simultaneously tracks the fast equilibrium and the leakage preconditioner online, achieving a remarkable T^{-1/2+o(1)} convergence rate with only O(1) primitive samples per iteration. This brings theoretical performance closer to the optimal rates seen in simpler stochastic gradient methods.

The work provides both a theoretical ceiling on naive approaches and a practical roadmap for acceleration, with implications for any application where two-time-scale stochastic approximation is used — from deep reinforcement learning policy evaluation to meta-learning and online convex optimization.