New math reveals how adversarial training fails under SGD with constant learning rates

Fabrizzio Sabelli's new paper on arXiv (2607.00207) provides a rigorous mathematical framework for understanding the learning dynamics of ℓ₂-adversarial training in high dimensions. The study focuses on single-index models trained on Gaussian mixtures under streaming stochastic gradient descent (SGD). The author derives deterministic equivalents for key statistics such as adversarial risk and distance to adversarial optimality, expressed as solutions to a system of ordinary differential equations (ODEs).

Two idealized learning rate schedules are examined: Polyak stepsize and exact line search. In the case of ℓ₂-adversarial least squares with a single class, surprisingly, no constant learning rate can guarantee monotone descent of SGD toward a minimizer of adversarial risk — unlike standard least squares. The paper identifies anisotropic covariance and a mismatch in ridge parameters as the root causes of exact line search's suboptimality relative to Polyak stepsize. Additionally, an SDE called adversarial homogenized SGD captures the evolving statistics of iterates, showing equivalence to standard least squares with adaptive learning rate and ℓ₂-regularization, up to dimension-free constants. When dynamics converge, the limiting adversarial risk and iterate are determined by a fixed-point equation, with the iterate equivalent to a ridge regression solution whose regularization parameter is the effective regularization from SGD.