ZEUS: An Efficient GPU Optimization Method Integrating PSO, BFGS, and Automatic Differentiation
New open-source algorithm tackles high-dimensional, non-convex problems by merging global and local search techniques.
A team from Old Dominion University and Fermilab has introduced ZEUS, a new computational method designed to tackle the notoriously difficult challenge of high-dimensional, non-convex optimization. The algorithm's innovation lies in its two-phase hybrid approach. First, it uses Particle Swarm Optimization (PSO), a population-based metaheuristic, to explore the problem space broadly and identify a set of promising candidate starting points. This global search phase helps avoid getting trapped in poor local minima.
In the second phase, ZEUS launches multiple, parallel runs of the highly efficient Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm from each of the starting points identified by PSO. A key feature is its integration of automatic differentiation (AD), which automatically computes the precise gradients BFGS requires, eliminating error-prone manual derivative calculations. By executing both phases on GPUs, ZEUS achieves substantial speedups, making it practical for complex real-world problems in fields like AI, where optimizing neural network parameters or loss landscapes is essential.
- Hybrid PSO-BFGS approach uses PSO for global search and BFGS for precise local optimization from multiple starting points.
- Integrates Automatic Differentiation (AD) to compute exact gradients for BFGS, removing manual derivative work for users.
- GPU-accelerated implementation provides significant performance gains for complex, non-convex problems common in machine learning.
Why It Matters
Provides a faster, more robust tool for optimizing complex AI models and scientific simulations, directly impacting research and development efficiency.