User Simulator-Guided Multi-Turn Preference Optimization for Reasoning LLM-based Conversational Recommendation

A team of researchers has introduced SMTPO (Simulator-guided Multi-Turn Preference Optimization), a novel framework designed to significantly improve AI-powered conversational recommendation systems. The core challenge these systems face is the "cold start" problem in multi-turn dialogues: without extensive real user data, AI models struggle to accurately learn and adapt to complex, evolving preferences. SMTPO tackles this by first creating a sophisticated AI user simulator. This simulator is trained via multi-task supervised fine-tuning (SFT) on diverse datasets, enabling it to generate high-quality, natural language feedback that better reflects a spectrum of potential user interests, even without explicit preference labels.

In the second stage, the framework trains the main LLM-based recommender. It first learns foundational reasoning patterns through SFT. Then, it engages in simulated multi-turn conversations with the AI user simulator. Crucially, the recommender is optimized using reinforcement learning (RL), where it receives fine-grained rewards for making suggestions that align with the simulator's inferred preferences. This RL process allows the recommender to progressively correct for any biases or errors in the simulator's feedback, preventing mistakes from compounding over a conversation. The researchers report that extensive testing on public datasets shows the method is both effective and transferable, meaning the trained models can generalize to new recommendation scenarios.