τ-Rec benchmark reveals AI agents fail 43% of recommendation tasks

Evaluating conversational AI agents that recommend movies, products, or content has become increasingly difficult as systems move from one-shot suggestions to multi-turn dialogues. Traditional benchmarks rely on 'LLM-as-a-judge' which introduces subjectivity, high cost, and inconsistency. To address this, researchers from an undisclosed institution (authors Bharath Sivaram Narasimhan and Karthik R Narasimhan) present τ-Rec, a benchmark that replaces subjective evaluation with verifiable rewards. Its key innovation is a 'reveal-tagged elicitation' (RTE) mechanism that controls how task constraints surface during conversation, and a 'pass^k' metric that measures how consistently an agent reasons across multiple attempts. The benchmark tests agents against structured catalog predicates rather than human ratings.

The team evaluated nine configurations across six models from five families: GPT-5.4, Claude Sonnet 4.6, Gemini 2.5 Flash, DeepSeek V4 Flash, Qwen3-32B, and GPT-5 mini. Results show a dramatic 'reliability cliff': even the top model only reached ~57% accuracy at pass^1 (single attempt) and dropped to ~38% at pass^4 (needing all four attempts correct). This indicates that current agentic recommender systems lack consistent reasoning ability in multi-turn interactions, a critical flaw for real-world deployment in customer service, e-commerce, or content curation. All code and data are open-source at the provided URL.