New Closed-Form Model Predicts GRPO Training Dynamics with 91% Accuracy

Group Relative Policy Optimization (GRPO) has become a standard method for improving the reasoning ability of large language models, but until now its training dynamics were understood only empirically – reward curves were fit with low-parameter functions that carried no mechanistic meaning, and hyperparameters were chosen through costly trial and error. In this new paper, a team of researchers introduces a first-principles reduced-order model that changes everything. Their closed-form model subsumes the empirical single-exponential saturation law as its overdamped limit, recasting the fitted plateau, timescale, and size exponent as fixed points of an underlying potential. The model also adds a slow-start phase that no single-exponential representation can capture. Crucially, all predictions are tied to independently measurable quantities rather than fitted constants: group-size invariance of the deterministic trajectory with a 1/G stationary fluctuation, a sharp stability threshold in the refresh interval, and a clear overdamped-to-oscillatory transition.

Across three different LLMs and two group sizes, the closed-form trajectory fits training reward with R² ≥ 0.91, and the predicted group-size invariance holds not only on the reward curve but also on out-of-distribution transfer to eight math benchmarks. The model also provides diagnostic tools that separate failure modes a reward curve alone would conflate – reward hacking, advantage degeneracy, policy concentration, and dynamical instability. In a controlled softmax-bandit reduction where the mean-field assumption holds exactly, the authors reproduce the predicted overdamped-to-oscillatory transition and locate the refresh-interval stability threshold at the independently measured stiffness. A full deep-network demonstration is left to future work, but this theoretical foundation already gives practitioners a framework to optimize GRPO training without black-box guessing.