VP2O: New RLHF method boosts Codeforces ELO by 179, cuts tokens 32%

Reinforcement Learning from Human Feedback (RLHF) via Proximal Policy Optimization (PPO) has been a cornerstone for aligning large language models, but it suffers from policy mode collapse, brittle exploration loops, and distribution drift. A new paper from Ousmane Amadou Dia introduces Variational Proximal Policy Optimization (VP2O), a particle-based variational inference framework that reimagines policy optimization using Stein Variational Gradient Descent (SVGD) within a Mixture-of-Experts (MoE) architecture. Instead of relying on fixed clipping or KL schedules, VP2O employs functional kernels over localized expert prototypes alongside an expert orthogonalization loss, creating a geometry-based proximal-control mechanism that stabilizes training and encourages diverse exploration.

Tested on a 33B/4B sparse MoE model, VP2O delivers impressive results: a +179 ELO gain on Codeforces coding benchmarks and a 32% reduction in token count on AIME mathematical reasoning tasks. These improvements come from better exploration and reduced distribution drift, enabling the model to produce more efficient reasoning chains. The paper demonstrates that moving beyond heuristic clipping to theoretically grounded variational inference can significantly improve both the stability and sample efficiency of RLHF. This could lower the computational cost of aligning future models while achieving stronger performance, especially in complex reasoning domains.