Gradient Smoothing: New optimizer trick boosts all deep learning models
Simple layer-wise smoothing improves LLMs, diffusion, and ViTs with zero overhead.
A new paper introduces Depth-wise Gradient Augmentation, a general optimization paradigm. Within this framework, Gradient Smoothing is studied, a family of depth-wise smoothing methods instantiated with a simple local Window Smoothing operator. The method operates on block-wise updates from arbitrary base optimizers (e.g., SGD, Adam, Muon), incurs minimal computational overhead, and is compatible with existing pipelines. Evaluated on language model pretraining, RL post-training of LLMs for reasoning, diffusion modeling, and image classification with Vision Transformers, it consistently improves optimization and generalization without modifying model architectures or training objectives.
- Gradient Smoothing works with SGD, Adam, and Muon optimizers with minimal overhead.
- Tested on LLM pretraining, RL reasoning, diffusion models, and ViTs — all showed improvement.
- Acts as a structured depth-wise preconditioner, promoting consistent representation evolution.
Why It Matters
A simple, universal optimization improvement that works across all modern deep learning architectures with zero extra cost.