MOON algorithm uses matrix geometry to improve multitask model training

Multi-task learning often relies on multi-objective optimization (MOO) to resolve conflicts between task gradients. However, existing approaches flatten model parameters into vectors and operate under Euclidean geometry, ignoring the natural matrix structure of modern architectures like Transformers. In a new arXiv paper, Shiji Zhou and colleagues introduce MOON (Multi-Objective OrthoNormalized Updates), which instead performs gradient manipulation under spectral-nuclear norm geometry—a choice that aligns with the theory of steepest descent for matrix-valued parameters. This allows the method to compute orthonormalized manipulated gradients that respect parameter structure, producing more effective update directions.

MOON provides theoretical guarantees for smooth non-convex objectives, establishing convergence of the averaged Pareto-stationarity measure at rates of O(T^{-1/2}) in deterministic settings and O(T^{-1/4}) under stochastic gradients. Empirically, the method consistently improves both optimization efficiency and final multi-task performance across multiple benchmark tasks. The authors have released code, enabling researchers to adopt the approach directly. MOON is particularly relevant as large-scale multi-task training becomes standard, suggesting that matrix-aware gradient geometry could be a simple and effective upgrade over current Euclidean-based MOO techniques.