Regularized Online RLHF with Generalized Bilinear Preferences

A team of researchers has published a significant theoretical advance in reinforcement learning from human feedback (RLHF) with their paper 'Regularized Online RLHF with Generalized Bilinear Preferences.' The work tackles the core problem of contextual online RLHF with the goal of identifying Nash Equilibrium, moving beyond the limitations of standard preference models. The authors introduce the Generalized Bilinear Preference Model (GBPM), a novel framework that uses low-rank, skew-symmetric matrices to capture complex, potentially intransitive human preferences—where choice A may be preferred to B, B to C, but C to A. Crucially, their analysis generalizes prior work by supporting any strongly convex regularizer, not just reverse KL-divergence. A key theoretical insight proves that the dual gap of a greedy policy is bounded by the square of the estimation error, a result derived from the strong convexity and skew-symmetry of their model.

Building on this foundation and a feature diversity assumption, the paper establishes two groundbreaking regret bounds via simple, practical algorithms. The first, Greedy Sampling, achieves a polylogarithmic regret of Õ(ηd⁴(log T)²), which is notably free from exponential dependence on the regularization strength (e^(O(η))). The second, Explore-Then-Commit, exploits the low-rank (rank r) structure of the GBPM to achieve a regret of Õ(√(ηrT)). This latter result is the first statistically efficient, dimension-free (poly(d)-free) guarantee for online RLHF in high-dimensional settings, a major milestone for scaling alignment techniques. The 43-page work provides a robust mathematical framework that could enable more efficient and stable training of large language models like GPT-4 or Claude by better modeling nuanced human feedback during online interaction.