Shock-wave theory reveals hidden structure in neural network training dynamics

A new theoretical framework by Taiki Miyagawa, accepted to ICANN 2026, establishes a rigorous connection between shock-wave theory and the learning dynamics of artificial neural networks under stochastic gradient descent. The key innovation is to quotient out parameter symmetries (redundancies in the weight space) and apply local-entropy coarse-graining. On the resulting quotient manifold, the effective dynamics satisfy a viscous Hamilton–Jacobi equation. Further, if the raw parameter dynamics are summarized by a gradient field on the quotiented space, the gradient of the coarse-grained loss obeys a Burgers-type equation—a classic PDE from fluid mechanics that describes shock formation. The paper rigorously proves that shock waves can form in the learning landscape under these conditions.

The theory is validated across multiple architectures: multilayer perceptrons, convolutional neural networks, Transformers, and mean-field networks all exhibit the predicted Hamilton–Jacobi or Burgers-type behavior. The practical implication is significant: in large models like Transformers, raw parameter norms are often distorted by symmetry redundancy and can mislead training diagnostics. By using symmetry-corrected quotient observables, practitioners gain a principled basis for monitoring, forecasting, and controlling phase transitions during training—potentially enabling early warning of training collapses or sudden improvements.