OrthoGrad curbs noisy-label memorization — but only in small data regimes
A new gradient projection technique reduces overfitting on corrupted labels... sometimes.
In new research accepted at ICML HiLD 2026, Richard Mai proposes OrthoGrad, a geometric intervention on the optimizer that subtracts the component of each weight gradient parallel to the current weight vector. The idea is to constrain updates to the orthogonal subspace, potentially discouraging memorization of noisy labels. On MNIST with small-data settings, OrthoGrad notably improved test accuracy for CNNs while reducing the fitting of corrupted labels. Mechanism analysis revealed that the effect is strongest when raw gradients contain a nontrivial radial component — i.e., when weights and gradients are not already orthogonal.
However, scaling up to CIFAR-10 with ResNet-18 told a different story. OrthoGrad altered the trajectory of memorization but did not prevent eventual overfitting to noisy labels. In larger-data regimes where gradients are naturally nearly orthogonal to weights, the projection has little effect. Mai concludes that OrthoGrad is best used as a diagnostic tool for studying learning dynamics rather than as a universal regularizer. The paper underscores how geometric constraints interact with data scale and architecture.
- OrthoGrad removes the radial gradient component, reducing corrupted-label fitting on small-data MNIST with CNNs.
- The method's effect is strongest when raw gradients have a large component parallel to weight vectors.
- On CIFAR-10 ResNet-18, OrthoGrad delayed but did not prevent eventual memorization of noisy labels.
Why It Matters
Provides a new geometric lens on overfitting, but shows that data size still dominates regularization.