New RELTA-SGLD method stabilizes nonconvex ML training with 2x accuracy gains
Tames superlinear stochastic gradients while preserving nonvanishing tail returns...
Researchers Yiwei Zhou and Ziheng Chen introduced RELTA-SGLD, a novel taming scheme for nonconvex Stochastic-Gradient Langevin Dynamics (SGLD). The method addresses a critical issue in modern ML training: superlinear stochastic-gradient oracles can cause instability, while existing taming methods often suppress useful learning drift. RELTA uses a localized threshold to decide when to activate taming, and a relative-growth principle derived from one-step Lyapunov stability to determine the required strength. This produces a lighter λ-scale denominator that preserves far-tail returns, enabling the algorithm to maintain near-original dynamics in ordinary training scenarios.
The paper proves that RELTA achieves polynomial moment stability and first-order stationary accuracy in both Wasserstein-1 and Wasserstein-2 distances, improving over comparable tamed SGLD schemes by half-order and quarter-order bounds. Empirically, on Fashion-MNIST under active stabilization pressure, RELTA improved mean learning metrics over both untamed SGLD and the TUSLA baseline, staying competitive with a tuned AdamW reference. The method's lighter localized denominator reduces unnecessary perturbation, making it practical for both aggressive and ordinary training regimes. Code is available, and the 40-page paper provides detailed theoretical and experimental analysis.
- Uses a relative-growth principle for adaptive taming, improving stability bounds by half-order to quarter-order over existing methods.
- On Fashion-MNIST, RELTA outperforms untamed SGLD and TUSLA, matching a tuned AdamW in mean learning metrics.
- Preserves nonvanishing far-tail returns, enabling better exploration in nonconvex optimization landscapes.
Why It Matters
A theoretically grounded optimizer that stabilizes training without sacrificing accuracy, potentially improving deep learning workflows.