New paper identifies weight-norm criticality as cause of training loss spikes

A new paper from researchers Xiaolong Li, Zhangchen Zhou, and Zhi-Qin John Xu (arXiv:2607.21005) introduces the concept of weight-norm criticality to explain training loss spikes in deep neural networks. While prior work focused on learning-rate criticality (Edge of Stability), this work highlights a separate mechanism: the interaction between normalization layers (which create scale-invariant weights) and weight decay (which persistently shrinks parameter norms). As the weight decay coefficient increases, the norms of scale-invariant weights are driven toward zero, causing the sharpness of the loss landscape to rise rapidly. This destabilizes optimization and produces abrupt loss spikes—even when learning rates remain moderate.

The authors argue that this provides a new, mechanistic understanding of why weight penalties improve generalization but cannot be made arbitrarily strong: excessive decay pushes weight norms past a critical boundary, triggering instability. The paper offers testable predictions, which are empirically validated in networks with scale-invariant components (e.g., batch-normalized models). For practitioners, this means tuning weight decay is not just a regularization trade-off but a stability constraint—stronger decay risks catastrophic loss spikes. This work bridges a gap between theory and practical training instabilities, offering guidance for robust hyperparameter selection.