Fisher-Geometric Diffusion in Stochastic Gradient Descent: Optimal Rates, Oracle Complexity, and Information-Theoretic Limits

Researchers Daniel Zantedeschi and Kumar Muthuraman have published a foundational paper, 'Fisher-Geometric Diffusion in Stochastic Gradient Descent,' which reframes our understanding of optimization noise. The core breakthrough is identifying that the covariance of mini-batch gradients in SGD is not arbitrary external noise but is intrinsically determined by the geometry of the loss function itself. Specifically, under exchangeable sampling, this covariance equals the projected Fisher information matrix for well-specified models or the projected Godambe (sandwich) matrix for general M-estimation. This forces a new diffusion approximation where the volatility has a Fisher/Godambe structure, characterized by an effective temperature τ = η/b (learning rate divided by batch size).

Building on this geometric foundation, the authors prove rigorous, matching minimax upper and lower bounds of order Θ(1/N) for the Fisher/Godambe risk, where N is the total oracle (gradient evaluation) budget. Crucially, their lower bound holds under a broad martingale oracle condition, subsuming standard i.i.d. sampling. This theory implies that convergence guarantees for reaching ε-stationarity depend on an intrinsic effective dimension and a Fisher/Godambe condition number, not the often-misleading ambient dimension or Euclidean conditioning. Experiments validate their derived Lyapunov equation for the stationary covariance and demonstrate that simpler scalar noise models fail to capture this critical directional structure. The work provides a unified information-theoretic limit for SGD, connecting optimization, statistics, and geometry.