New study reveals PRF harms 25% of searches; LLM audit fixes it
Only 20.9% of queries benefit from Pseudo-Relevance Feedback—researchers built a fix using LLMs.
Pseudo-Relevance Feedback (PRF) is a common technique to improve search relevance by expanding queries with terms from top-ranked documents. However, this paper reveals that PRF actually harms a significant fraction of queries—25.6% in their user study—due to query drift, a problem hidden by aggregate metrics. The researchers conducted a participatory audit with 108 users across 43 TREC Deep Learning 2019 queries, finding that avoiding harm from PRF is nearly twice as valuable as exploiting successful expansion.
To solve this, the team repurposes LLM-based rerankers as system preference predictors that automatically replicate user-derived labels. This two-stage framework—audit then automate—explains which queries PRF harms, why a selective PRF decision is made, and how it can be inspected at scale. The work, accepted at WExIR @ SIGIR 2026, turns an opaque retrieval component into an auditable, user-grounded one, with implications for search engines, enterprise retrieval, and any system using relevance feedback.
- Only 20.9% of queries benefit from PRF; 25.6% suffer degraded user experience (study of 108 users, 43 TREC queries).
- LLM-based rerankers automatically predict harmful PRF expansions with inspectable evidence, replacing manual auditing.
- Avoiding PRF harm is nearly twice as valuable as exploiting successful expansion, per user preference data.
Why It Matters
Makes query expansion transparent and safe—critical for search systems where PRF can silently degrade results for one in four queries.