New REINFORCE method optimizes generative retrieval for long-term user satisfaction
A multi-step RL approach beats next-item prediction on Yambda-5B by directly optimizing session-level rewards.
Researchers from Matveev et al. propose an off-policy REINFORCE method for large-scale generative retrieval that optimizes long-term user satisfaction rather than just next-item prediction. The approach uses multi-step importance weighting and a user feedback model for offline evaluation. On the Yambda-5B dataset, it improves offline estimates of cumulative session reward over next-item and next-positive prediction baselines while largely preserving retrieval quality. Additionally, allocating more inference-time compute to simulating future responses improves model-based long-term return estimates without updating the policy.
- Uses off-policy REINFORCE with multi-step importance weighting for session-level optimization, outperforming one-step corrections.
- Introduces a user feedback model and doubly robust evaluation for sequential recommendation, validated on the Yambda-5B dataset.
- Test-time scaling improves long-term returns by up to 15% without retraining the generative retriever policy.
Why It Matters
Provides a scalable RL approach to optimize recommendations for long-term engagement, directly applicable to large-scale platforms.