HRPO boosts AI recommendation systems with token-level utility optimization

Generative recommenders, which select items by autoregressively decoding semantic identifiers (SIDs), are typically trained with supervised next-token prediction on logged trajectories. This approach mimics past behavior rather than directly optimizing for downstream utility. To close that gap, the team behind HRPO introduces a post-training framework that transforms item-level rewards into token-level training signals, making credit assignment more granular and stable. HRPO first estimates prefix-level utilities using group-wise reward smoothing over feature-based user clusters, then decomposes them into residual token credits and aggregates them into credit-to-go signals. The final stage, Residual-Return Policy Optimization (RRPO), applies clipped updates, group-normalized advantages, and KL regularization to maintain policy stability.

In experiments, HRPO outperformed baselines on a public dataset and showed consistent gains in session-level utility and key business metrics in an online A/B test within a large-scale commercial system (Kuaishou's recommendation platform). This suggests the method is practical and scalable for real-world deployment. The authors also released source code and archived artifacts for reproduction. Building on earlier work like AlignRec and GRecX, the paper highlights the growing focus on aligning generative recommenders with direct utility metrics. For engineers building recommendation systems, HRPO offers a concrete, reusable approach to post-train generative models beyond simple imitation learning.