Evolution strategies beat RL for diverse LLM solutions, arXiv study shows
Post-training with ES yields broader solution coverage and higher pass@k than RL
In discovery domains like math and science, LLMs are typically used to produce a single best answer. But pass@k—where the model generates multiple candidate solutions—can improve outcomes if the candidates are diverse. Standard post-training with Reinforcement Learning (RL) narrows the model's output distribution toward high-reward responses, causing solution coverage to collapse: the model keeps rehashing similar ideas instead of exploring the space.
The paper, authored by Conor F. Hayes, Elliot Meyerson, Kajetan Schweighofer, Roberto Dailey, Babak Hodjat, Risto Miikkulainen, and Xin Qiu, proposes an alternative: Evolution Strategies (ES). ES is a population-based, gradient-free method that perturbs the model's weights directly and selects for perturbations that improve performance. Across standard math benchmarks, ES post-training consistently achieves higher pass@k than RL while maintaining a broader output distribution. This means LLMs trained with ES explore more diverse solution paths, a critical advantage for scientific and mathematical discovery where creative alternatives matter more than converging on one answer.
- ES post-training achieves consistently higher pass@k than RL across math benchmarks
- Optimizes directly in weight space via random perturbations—no gradients needed
- Produces broader output distributions, preventing solution coverage collapse from RL-style narrow convergence
Why It Matters
For AI-driven scientific and mathematical discovery, diverse candidate solutions are key; ES post-training could substantially improve LLM exploration beyond single best guesses.