FlowAdam optimizer cuts error 22% with ODE-based momentum injection

FlowAdam, developed by Devender Singh and Tarun Sheel, tackles a known weakness of adaptive optimizers like Adam: their diagonal, coordinate-wise preconditioners treat each parameter independently, struggling with dense or rotated parameter couplings common in matrix factorization, tensor decomposition, and graph neural networks. The new method augments Adam with continuous gradient-flow integration through an ordinary differential equation (ODE). When exponential moving average statistics detect landscape difficulty, FlowAdam switches to clipped ODE integration. Its central innovation, Soft Momentum Injection, blends ODE velocity with Adam's momentum during mode transitions, preventing the training collapse that plagues naive hybrid approaches.

Empirically, FlowAdam delivers meaningful gains on coupled optimization benchmarks: 10-22% lower held-out error on low-rank matrix and tensor recovery, and 6% improvement on the Jester real-world collaborative filtering dataset. It also surpasses tuned Lion and AdaBelief optimizers while matching Adam's performance on well-conditioned workloads like CIFAR-10. Ablation studies confirm the soft injection is essential—hard replacement drops accuracy from 100% to 82.5%. The work is accepted at IJCNN 2026 (IEEE WCCI) and marks a practical step toward geometry-aware optimization for structured machine learning problems.