Sven: Singular Value Descent as a Computationally Efficient Natural Gradient Method
New optimization method uses truncated SVD to achieve natural gradient performance without quadratic scaling.
A team from MIT and CERN has introduced Sven (Singular Value dEsceNt), a breakthrough optimization algorithm that rethinks how neural networks learn. Instead of reducing the entire loss function to a single scalar before computing updates—the standard approach in methods like stochastic gradient descent—Sven treats each individual data point's residual as a separate condition to be satisfied simultaneously. It finds the minimum-norm parameter update that best satisfies all conditions using the Moore-Penrose pseudoinverse of the loss Jacobian, approximated via a computationally efficient truncated singular value decomposition (SVD).
This architecture gives Sven the mathematical properties of natural gradient methods—which adjust updates according to the geometry of the loss landscape—but without their crippling computational overhead. Traditional natural gradient approaches scale quadratically with the number of parameters, making them impractical for modern large models. Sven's truncated SVD retains only the k most significant directions, resulting in linear scaling (a factor of k) relative to standard SGD. In practical tests on regression tasks, Sven significantly outperformed first-order optimizers like Adam, converging faster to lower final losses, while remaining competitive with second-order methods like LBFGS at a fraction of the wall-time cost.
The primary challenge for adoption is memory overhead, as the method requires storing and processing the Jacobian matrix. The authors propose mitigation strategies and highlight that Sven is particularly well-suited for scientific computing applications where custom loss functions naturally decompose into multiple conditions. This makes it promising for fields like high-energy physics, where the research team has direct expertise. As neural networks grow larger and training costs skyrocket, efficient optimization algorithms like Sven could substantially reduce the computational burden of developing advanced AI models.
- Uses truncated SVD to approximate natural gradient updates with only linear (k-factor) computational overhead vs. quadratic scaling
- Outperforms Adam on regression tasks, converging faster to lower loss, and matches LBFGS performance at reduced cost
- Particularly suited for scientific computing where loss functions decompose into multiple conditions, with applications in physics
Why It Matters
Could dramatically reduce training costs for large neural networks while improving convergence, especially in scientific domains.