Adaptive Optimization via Momentum on Variance-Normalized Gradients

Researchers have introduced MVN-Grad, a new Adam-style optimizer that reportedly outperforms current standards like Adam and AdaBelief. By applying momentum after variance normalization, it decouples stale momentum from stochastic normalization, reducing one-step update variance by up to 30% and providing robustness against outlier gradients. In tests on CIFAR-100 and GPT-style language modeling, it delivered smoother training and improved generalization with no added computational overhead, matching or beating benchmarks.