A Systematic Study of Pseudo-Relevance Feedback with LLMs

A new research paper from Nour Jedidi and Jimmy Lin provides the first systematic analysis of how to effectively implement Pseudo-Relevance Feedback (PRF) with Large Language Models. PRF is a technique where an initial search query is automatically refined using feedback from top-ranked documents, but implementations vary widely. The study isolates two key design dimensions: the feedback source (where the text comes from) and the feedback model (how that text refines the query). Through controlled experiments across 13 low-resource BEIR benchmark tasks and five different LLM PRF methods, the researchers were able to disentangle the impact of each dimension.

The results offer clear, actionable guidance for engineers. First, the choice of feedback model itself plays a critical role in overall effectiveness, meaning the algorithm used to process the feedback is as important as the data. Second, and perhaps most surprisingly, feedback text generated entirely by an LLM (like GPT-4 or Claude) provides the most cost-effective solution, reducing dependency on external corpus data. However, the third finding adds nuance: when using actual corpus documents as the feedback source, the technique is most beneficial when those documents come from a strong first-stage retriever (like a dense vector search).

This research moves beyond anecdotal evidence to provide a principled framework for building better retrieval systems. For teams working on search engines or RAG (Retrieval-Augmented Generation) pipelines, these findings help prioritize development efforts. Instead of guessing which aspect to optimize, developers can now focus on selecting the right feedback model and strategically choosing between LLM-generated or corpus-sourced feedback based on their system's retrieval strength and cost constraints.