Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space

Researcher Zixian Wang has introduced a novel theoretical framework for aligning large language models (LLMs) called Group Orthogonalized Policy Optimization (GOPO). Published on arXiv, the work fundamentally reframes the reinforcement learning from human feedback (RLHF) problem. Instead of optimizing policies on the traditional probability simplex—which inherits the exponential curvature of the Kullback-Leibler divergence—GOPO lifts the entire alignment process into an infinite-dimensional Hilbert space of square-integrable functions. Within this L2(pi_k) space, the complex constraint that probabilities must sum to one simplifies to a linear orthogonality condition, transforming a nonlinear optimization into a geometric projection problem.

The core innovation is applying the Hilbert projection theorem to find the policy closest to an unconstrained target, resulting in a 'work-dissipation' functional with a known maximizer. A key practical outcome is a closed-form threshold that induces exact sparsity, automatically assigning zero probability to catastrophically poor actions without heuristic clipping. By projecting from the infinite-dimensional theory to a finite empirical subspace via group sampling, GOPO derives a practical training objective with a constant Hessian (mu I), linear gradients, and a built-in 'dead-zone' mechanism. Initial experiments on mathematical reasoning benchmarks indicate GOPO achieves competitive generalization while maintaining stable gradient dynamics and preserving policy entropy in regimes where standard clipping-based methods like PPO plateau.