The Theory and Practice of Highly Scalable Gaussian Process Regression with Nearest Neighbours

A team of researchers has published a comprehensive 92-page paper that finally provides a complete theoretical framework for scalable Gaussian Process (GP) regression using nearest neighbours. The work, titled 'The Theory and Practice of Highly Scalable Gaussian Process Regression with Nearest Neighbours,' tackles the long-standing problem of GP's cubic computational complexity, which has limited its application on massive datasets. The authors develop rigorous proofs for the statistical properties of Nearest Neighbour Gaussian Process (NNGP) and the related GPnn method, which have shown strong empirical performance but lacked complete theoretical justification.

Under mild regularity assumptions, the researchers derive almost sure pointwise limits for key predictive criteria: mean squared error (MSE), calibration coefficient (CAL), and negative log-likelihood (NLL). They prove universal consistency and demonstrate that the method's L2-risk achieves Stone's minimax rate of n^{-2α/(2p+d)}, where α and p capture the regularity of the regression problem. This establishes NNGP/GPnn as statistically sound alternatives to full GP models.

The paper also proves uniform convergence of MSE over compact hyper-parameter sets and shows that its derivatives with respect to lengthscale, kernel scale, and noise variance vanish asymptotically with explicit rates. This mathematical explanation accounts for the observed robustness of GPnn to hyper-parameter tuning in practice, addressing a key practical concern for machine learning practitioners who need reliable, scalable regression tools.