Why is Normalization Preferred? A Worst-Case Complexity Theory for Stochastically Preconditioned SGD under Heavy-Tailed Noise
A major paper reveals why your optimizer might be failing to converge.
Deep Dive
A new theoretical paper provides a worst-case complexity theory for popular adaptive optimizers like Adam and RMSProp under heavy-tailed noise. It proves normalization (common in Adam) guarantees convergence at optimal rates, while clipping can fail entirely in the worst case due to statistical dependencies. This offers the first rigorous explanation for the empirical preference for normalization in large-scale model training, settling a key debate in optimization.
Why It Matters
This provides the theoretical backbone for choosing optimizers, potentially leading to more stable and faster training of large AI models.