Why AI Model Weights Grow, Overshoot, Then Relax — The Hidden Weibull Dynamics Under AdamW Training

A new paper by Tiexin Ding, published on arXiv, dives into the mechanics of weight evolution during neural network training. Specifically, it studies the scale parameter λ of a Weibull distribution fitted to transformer weight distributions under the popular AdamW optimizer. The central finding is a three-force decomposition of the squared weight norm: an alignment force (correlation between weights and adaptive updates), an injection force (adaptive step magnitude), and a decay force (from decoupled weight decay). On self-trained Pythia-70M models with full optimizer trace, alignment accounts for 88–94% of the absolute force budget across random seeds. Near saturation, alignment and decay roughly balance, explaining the characteristic peak and relaxation of λ(t).

To extend the analysis to realistic training setups (where optimizer moments aren't saved), Ding introduces a spline displacement method that reconstructs alignment force from sparse checkpoints with ~92–94% accuracy—nearly double that of a naive two-point baseline. The paper also observes that the peak value of λ varies with training-data coherence, hinting at a data-dependent component of weight-scale growth slated for future study. The work provides concrete tools and metrics for diagnosing training dynamics, and all code and data are publicly available. This research offers a mathematical lens for understanding why transformer weights evolve the way they do under adaptive optimizers, potentially guiding better training regimes.