New ML theory proves Expected Improvement is optimally efficient
French researchers mathematically prove Expected Improvement (EI) achieves near-optimal convergence in Bayesian optimization.
French researchers Emmanuel Vazquez and Sébastien Petit have published a groundbreaking paper in arXiv that mathematically proves the Expected Improvement (EI) policy achieves near-optimal convergence rates in Bayesian optimization. The work, titled 'Simple-regret rates and minimax optimality of fixed-prior expected improvement in Matérn and squared-exponential RKHSs,' rigorously analyzes EI's behavior when optimizing deterministic objective functions within Reproducing Kernel Hilbert Spaces (RKHS).
The paper demonstrates that for isotropic Matérn kernels with smoothness parameter ν>0, EI achieves simple regret of O(N^(-ν/d)) after N post-initial queries, where d is the input dimension. For the isotropic squared-exponential kernel, EI achieves exponential regret bounds of the form O(exp[-c₁min{N, N^(1/d)}log(eN)]), which improves to O(exp[-c₂N^(1/d)log(eN)]) with exact EI maximization. Crucially, these bounds are uniform over RKHS balls of any fixed radius B, proving EI's robustness across different function classes.
- Proves Expected Improvement (EI) policy achieves minimax-optimal convergence for Matérn kernels and near-optimal for squared-exponential kernels
- Derives O(N^(-ν/d)) regret for Matérn kernels and exponential regret bounds for squared-exponential kernels
- Results hold uniformly over RKHS balls of any fixed radius B, demonstrating theoretical guarantees across function classes
Why It Matters
Provides mathematical proof that Bayesian optimization's Expected Improvement is theoretically optimal, guiding more efficient hyperparameter tuning and black-box optimization.