AI agents fail when users lack preferences, new study finds

A new research paper from Irena Saracay, Ludwig Schmidt, and Carlos Guestrin challenges a core assumption in AI agent design: that users have well-formed preferences from the start. The authors introduce CoPref, a model based on the Search-Experience-Credence framework from Information Economics, which formalizes how users construct preferences through dialog with an agent. The key insight: agents must help users gain domain knowledge before they can meaningfully state what they want, rather than just asking clarifying questions.

To test these ideas, the researchers created CoShop, an interactive benchmark where an agent converses with a simulated user (CoPref) to make recommendations. When evaluating five frontier models, no agent achieved more than 56% accuracy over five dialogue turns. The failures were not due to search limitations but to the agent's inability to expand the user's knowledge about features and options. The study underscores that for real-world applications, agents must shift from eliciting preferences to actively educating users, a finding with significant implications for customer service, e-commerce, and AI assistants.