EvoPref: Evolutionary optimization beats gradient descent for diverse LLM alignment

A team led by Dongxin Guo, Jikun Wu, and Siu Ming Yiu from HKU has introduced EvoPref, a multi-objective evolutionary algorithm designed to overcome a critical flaw in current LLM alignment methods: preference collapse. Traditional gradient-based approaches like DPO, IPO, KTO, and ORPO tend to converge on narrow behavioral modes, sacrificing diversity for single-objective performance. EvoPref maintains a population of Low-Rank Adaptation (LoRA) adapters and optimizes them across three key objectives—helpfulness, harmlessness, and honesty—using Non-dominated Sorting Genetic Algorithm II (NSGA-II) with archive-based diversity preservation.

On standard benchmarks, EvoPref significantly outperforms gradient baselines: it achieves a median preference coverage of 82.5% compared to 70.0% for ORPO (p<0.001, n=30), and cuts collapse rates by 47% (11.0% vs. 20.6%). Notably, this diversity gain does not come at the cost of alignment quality—EvoPref scores 75.5% on RewardBench, statistically comparable to ORPO's 75.0% (p<0.05). The authors provide theoretical motivation extending recent MOEA runtime analysis (Dang et al., 2025), explaining why archive-based methods escape collapse more effectively. Comprehensive comparisons against MOEA/D, SMS-EMOA, CMA-ES, and gradient baselines, with rigorous statistical testing (Friedman with Holm correction, Vargha-Delaney effect sizes), confirm that multi-objective selection with diversity preservation is the key driver. Accepted to GECCO 2026, this work establishes evolutionary optimization as a principled paradigm for diverse LLM alignment, offering a promising alternative to the current single-trajectory gradient descent approaches.