Walmart's LLM-annotated pipeline boosts sponsored search retrieval by 5.1% NDCG
Replaces click-based training with 240M+ LLM-labeled examples for e-commerce search
Walmart researchers have developed a scalable method for training dense retrieval models without click signals or manual annotation, using LLM-annotated structured supervision. Their pipeline, presented at SIGIR 2026, combines multi-channel retrieval mining from three production systems (lexical, vector, and hybrid), graded-relevance annotation via a calibrated three-model cascade (reaching 89.1% agreement with trained human annotators), and a progressive curriculum that organizes over 240 million training examples across five difficulty levels. The resulting two-tower BERT model was deployed on Walmart's sponsored search engine.
The system dramatically outperformed the click-trained production baseline. On 30K human-labeled queries, it achieved a +5.1% improvement in NDCG@10, with the largest gains on tail (rare) queries. The rate of embarrassing retrievals (relevance rating 0) fell from 8.7% to 3.5%. In a two-week online A/B test with tens of millions of ad requests per arm, the model drove a +2.80% increase in ad spend, +1.4% CTR, +2.8% eCPM, and +2.9% click conversion rate. This work provides a practical blueprint for replacing click-based supervision with structured LLM-annotated training in production retrieval systems.
- Three-model LLM cascade achieved 89.1% agreement with human annotators on query-item relevance
- Progressive curriculum training used 240M+ examples across 5 difficulty levels, improving tail query performance significantly
- Online A/B test showed +2.80% ad spend, +1.4% CTR, +2.8% eCPM, and +2.9% conversion rate
Why It Matters
Replaces biased click signals with scalable LLM supervision, delivering measurable revenue lift in production e-commerce search.