Boundary Bias in Gaussian Processes Skews Bayesian Optimization Exploration

Gaussian processes with stationary kernels are widely used in Bayesian optimization, but a long-known artifact—inflated posterior variance near domain boundaries—receives fresh scrutiny in a new preprint from researchers at Uppsala University. The authors trace the root cause to a simple geometric mechanism: when the domain is bounded, the kernel's correlation neighborhood is truncated at the edge, creating an observation-independent distortion that worsens significantly as dimensionality increases. This distortion arises without any reference to the objective function, meaning acquisition behavior can be dominated by kernel geometry rather than task-specific uncertainty.

The paper systematically examines how this bias manifests across three acquisition classes. Variance maximization concentrates selections at corners, whereas negative integrated posterior variance and expected predictive information gain move selections inward to axis-aligned interior shells. To help practitioners diagnose and compensate, the authors introduce a function-free selection-profile diagnostic that works for arbitrary acquisitions, kernels, and bounded-domain geometries. The findings have immediate implications for anyone using Gaussian processes in high-dimensional optimization tasks like hyperparameter tuning or experimental design—suggesting that naive acquisition might oversample boundaries and miss optimal interior regions.