New research proves SGD self-stabilizes with large learning rates
Stochastic Gradient Descent alternates between unstable and stable regimes but always returns to stability.
Modern deep learning routinely operates at the 'edge of stability'—using learning rates far larger than classical optimization theory allows. While prior analyses focused on deterministic gradient descent, this new work from Emmanouilidis et al. tackles the stochastic setting. The authors prove that when applying SGD to linear classifiers and two-layer neural networks with cross-entropy loss, the dynamics naturally oscillate between an edge-of-stability regime dominated by curvature-driven oscillations and a stable regime where the expected loss decreases controllably. Crucially, they show that SGD self-stabilizes: even with large learning rates, the iterates return to stability in a fixed number of steps, ensuring convergence in the best-iterate sense.
The theoretical guarantees are complemented by experiments that validate the findings and illustrate the practical benefits of the large-stepsize regime. This explains why practitioners can successfully use high learning rates without catastrophic divergence: the inherent stochasticity of SGD acts as a stabilizing force. The work opens the door to more aggressive training schedules and deeper understanding of the optimization dynamics underpinning modern deep learning.
- Proves convergence guarantees for SGD on multiclass cross-entropy with large learning rates for linear classifiers and two-layer networks.
- Shows dynamics alternate between curvature-driven unstable oscillations and stable loss decrease, with guaranteed return to stability in fixed iterations.
- Experiments confirm self-stabilization allows safe use of large learning rates, validating common practice in deep learning training.
Why It Matters
Explains why large learning rates work in practice, enabling faster convergence and more robust training of deep models.