Shopping Reasoning Bench reveals AI assistants still lack expert-level shopping advice

A new benchmark from researchers (Shuxian Fan et al.) dubbed the Shopping Reasoning Bench aims to fill a critical gap in evaluating conversational shopping assistants. Unlike existing general-purpose or e-commerce benchmarks, this one focuses on open-ended multi-turn reasoning that requires domain expertise—balancing subjective preferences, budget constraints, and cross-product trade-offs. The benchmark consists of 525 expert-authored missions (232 single-turn, 293 multi-turn) with 10,863 importance-weighted binary rubrics. These rubrics are organized under a taxonomy of five reasoning categories and fifteen subcategories covering preference refinement, trade-off analysis, and compatibility assessment.

Evaluation of nine models across three families (GPT, Claude, Gemini) revealed significant shortcomings. Overall pass rates ranged from 57% to 77%. On multi-turn missions, all models scored 13–29 points lower on optional above-and-beyond criteria than on required ones. Performance also degraded by 4–18 points as conversations progressed, indicating that current models handle basic requests but fall short when providing expert-level advice over extended interactions. The authors position this benchmark as a challenging testbed for future development of shopping assistants that serve hundreds of millions of customers.