Evolutionary Optimization Trumps Adam Optimization on Embedding Space Exploration

A team of researchers has demonstrated that a gradient-free evolutionary optimization strategy can outperform the widely-used Adam optimizer for a key task in AI image generation. In their paper "Evolutionary Optimization Trumps Adam Optimization on Embedding Space Exploration," Domício Pereira Neto, João Correia, and Penousal Machado applied the Separable Covariance Matrix Adaptation Evolution Strategy (sep-CMA-ES) to optimize the prompt embeddings for the Stable Diffusion XL Turbo model. This process, known as inference-time control, allows users to guide image generation toward specific objectives—like better aesthetic quality or closer alignment to a text prompt—without the computational burden of fine-tuning the model's millions of weights.

The researchers tested the method on 36 prompts from the Parti Prompts (P2) benchmark, evaluating outputs with a combined score from the LAION Aesthetic Predictor V2 and CLIPScore. Across three different weighting scenarios (favoring aesthetics, alignment, or a balance), the evolutionary sep-CMA-ES approach consistently achieved higher objective values than Adam. The study also analyzed the divergence from baseline images and reported on compute and memory footprints, suggesting sep-CMA-ES is not only more effective but also a practical alternative. This work highlights a potentially more efficient pathway for users and developers to achieve precise control over diffusion model outputs, moving beyond prompt engineering alone.