MLE brittleness in Gaussian Processes: new method beats tabular foundation models

A new paper from researchers at UC Irvine — Tyler R. Johnson, Kian Ben-Jacob, Christopher P. Muller, and Ramin Bostanabad — challenges a common assumption in machine learning: that maximum likelihood estimation (MLE) is a safe, reliable way to train Gaussian Processes (GPs). The study, posted on arXiv (2608.13793), systematically demonstrates that MLE is brittle when its underlying assumptions don't hold, which is often the case in real-world engineering design data. Even GPs, widely presumed to be robust against overfitting, can generalize poorly under MLE, leading to unreliable probabilistic predictions.

The authors evaluate theoretically grounded metrics as alternatives to MLE and propose practical solutions for GP hyperparameter optimization. Their extensive experiments show that these solutions significantly improve GP robustness in downstream tasks like Bayesian optimization. Notably, they report that the resulting GPs can outperform tabular foundation models — the large transformer-based models typically used for tabular data — in prediction accuracy, uncertainty quantification, and inference cost. This is a strong claim, suggesting that carefully trained classical models still hold an edge over deep learning in certain settings.

The paper includes 26 pages, 10 figures, and 2 tables, and all contributions are publicly available via GitHub. For practitioners, this is a blueprint for building accurate, robust GPs without relying on brittle MLE — a practical upgrade for anyone using Bayesian optimization, probabilistic regression, or classification in engineering design workflows.