Decision-aware User Simulation Agent for Evaluating Conversational Recommender Systems
New framework models choice overload with users who actually change their minds
Conversational recommender systems (CRS) are increasingly evaluated using user simulators, but existing frameworks often produce unrealistically high acceptance probabilities because they fail to model the hesitation and decision deferral typical in real consumers. A team of researchers from National Taiwan University (Yuan-Chi Li, Li-Chi Chen, Sung-Yi Wu, Yu-Che Tsai, Shou-De Lin) introduces Hesitator, a theory-grounded user simulation framework that explicitly models human decision-making under choice overload.
Hesitator features a modular Decision Module that separates utility-based item selection from overload-aware commitment decisions. When an agent presents too many options, the simulated user may hesitate or defer, mimicking real-world behavior. Experiments across multiple simulation frameworks, domains, sales modes, and LLM backbones show that integrating the module consistently mitigates unrealistic behaviors as overload increases. The simulations also reproduce known behavioral patterns from psychological economics, validating the model's fidelity.
- Hesitator uses a Decision Module to split item selection from commitment, enabling realistic hesitation under choice overload
- Outperforms existing LLM-based simulators across multiple domains, sales modes, and backbone models
- Reproduces established psychological economics patterns, confirming theoretical grounding
Why It Matters
More realistic AI evaluations mean better conversational recommender systems that understand when to simplify choices.