ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning
New RL-trained shopping agent beats larger models by balancing product accuracy, persuasiveness, and tool efficiency.
A research team from Renmin University of China and Microsoft has introduced ChatShopBuddy, a new AI agent framework designed to create more reliable conversational shopping assistants. The core challenge they address is applying post-training reinforcement learning (RL) to optimize agents for the complex, multi-objective reality of online shopping, where success depends on factual accuracy, persuasive dialogue, and efficient use of tools like product databases. Their solution is a complete methodology featuring a new benchmark called SmartShopBench, which captures diverse shopping intents with a hierarchical evaluation system.
To train the agent effectively, the researchers developed two key innovations. First, Hierarchical Reward Modeling (HRM) structures mixed rewards—spanning objective metrics, subjective qualities, and process efficiency—using conditional gating that reflects their logical dependencies. Second, Dynamic Contrastive Policy Optimization (DCPO) enables efficient training by dynamically selecting trajectories based on reward and reasoning length, balancing final response quality with operational cost. In extensive experiments, their RL-trained ChatShopBuddy agent consistently outperformed larger models relying on generic reasoning, achieving superior stability rather than just higher performance peaks.
The work provides a valuable blueprint for applying RL to real-world conversational agents beyond shopping, where agents must satisfy interdependent and sometimes competing goals. By decomposing complex quality requirements into measurable levels and structuring rewards hierarchically, the method offers a path to more robust and trustworthy AI assistants that can handle nuanced, multi-turn interactions while remaining efficient.
- Uses new Hierarchical Reward Modeling (HRM) to structure interdependent objectives like product correctness and persuasiveness.
- Introduces Dynamic Contrastive Policy Optimization (DCPO) for efficient training that balances response quality with reasoning cost.
- Outperforms larger generic models on the new SmartShopBench benchmark, demonstrating more stable and reliable performance.
Why It Matters
Provides a practical RL framework for building trustworthy AI agents that must balance accuracy, persuasion, and efficiency in real-world tasks.