Study: AI Recommenders Fail at Step One — Finding the Right Options

Recommendation systems work in two stages. First, a search step gathers a pool of possible items. Second, an AI "reranker" decides the order you see them in. Most research tests only that second step — and rigs the game by guaranteeing the correct answer is already sitting in the pool. A new paper by Zhaohui Wang, accepted for presentation at the CIKM 2026 conference, shows this setup inflates quality scores by 92–95% across three Amazon datasets.

The reason is what the author calls a "recall ceiling." Recall is simply whether the right item made it into the pile at all. Across eight datasets in three different domains, real-world search only captured 2–19% of relevant items when pulling the top 100 candidates. That creates a hard ceiling: if the perfect product or song never enters the pool, no amount of clever reordering can ever put it in front of you. The AI is rearranging a deck that's missing most of its cards.

The researchers then threw everything at the problem. Prompt engineering (rewriting instructions to the AI). Much bigger models, up to 168 times larger. Sequential models that track your history. Supervised neural rerankers. LoRA fine-tuning, a lightweight way to customize an AI model. Hybrid retrieval, which blends text search with behavior data. Score-aware prompting. And fusing the AI with collaborative filtering — the classic "people like you also bought this" math. None of it significantly beat that plain old collaborative-filtering baseline. When they handed the AI the classic scores as a hint, it mostly just copied them back.

The practical lesson: in these low-recall settings, fixing your search step matters far more than buying a smarter reranker. The author proposes a new testing standard, the Recall-Aware Evaluation Protocol, which checks how good retrieval is before judging reranking. One honest caveat — real production systems with richer data and live user feedback may behave differently, and better rerankers could shine there.