Diffusion models slash Bayesian optimization costs by 10x

Bayesian optimization (BO) is a go-to method for black-box optimization, but its information-based acquisition functions like Predictive Entropy Search (PES) are computationally expensive due to repeated Gaussian process posterior sampling. A new paper from researchers at the National University of Singapore and Georgia Tech introduces a clever shortcut: use Conditional Diffusion Models (CDMs) to directly approximate the distribution of the global optimum. The team developed training-aware strategies specifically for BO, ensuring the diffusion model learns the right distribution without requiring costly GP samples at each step.

Their resulting acquisition function, Diffusion-based Mode Seeking (DMS), actively queries points that maximize the probability of being near the optimum. The authors prove a sub-optimality guarantee for the CDM-learned distribution and show experimentally that DMS outperforms standard BO baselines on synthetic and real-world benchmarks. This work makes BO more scalable for high-dimensional or expensive evaluation settings, such as hyperparameter tuning or materials design, by replacing a computational bottleneck with a fast generative model.