Inverse-Free Sparse Variational Gaussian Processes

A team of researchers from leading institutions has developed a practical method called Inverse-Free Sparse Variational Gaussian Processes that addresses a fundamental bottleneck in Gaussian Process (GP) modeling. Traditional sparse variational GP approximations rely on Cholesky decompositions of kernel matrices, which are computationally expensive and ill-suited to modern low-precision, massively parallel hardware like GPUs. While previous attempts created variational bounds using only matrix multiplications via auxiliary parameters, optimizing them with standard first-order methods proved challenging.

The breakthrough comes from a better-conditioned bound and a novel matmul-only natural-gradient update for the auxiliary parameter, which dramatically improves stability and convergence. The researchers also provide practical heuristics including step-size schedules and stopping criteria that make the optimization routine seamlessly integrate into existing workflows. Across regression and classification benchmarks, the method demonstrates three key advantages: it serves as a drop-in replacement in SVGP-based models (including deep GPs), recovers similar performance to traditional methods, and can outperform baselines when properly tuned.

This work, accepted to AISTATS 2026, represents a significant step toward making Gaussian Processes more practical for large-scale applications. By eliminating the need for matrix inversions and Cholesky decompositions, the method aligns GP training with the strengths of modern hardware architectures while maintaining the statistical rigor that makes GPs valuable for uncertainty quantification in machine learning applications.