Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage
A landmark study finds strong correlation between retrieval quality and the information coverage of AI-generated answers.
A team of researchers from Johns Hopkins University and the University of New Hampshire has published a significant study titled 'Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coverage.' The paper, available on arXiv, systematically investigates a critical but understudied question in AI: can the quality of the initial document retrieval stage reliably predict the information coverage of the final answer generated by a RAG system?
Through extensive experiments across two text RAG benchmarks (TREC NeuCLIR 2024 and TREC RAG 2024) and one multimodal benchmark (WikiVideo), the team analyzed 25 different retrieval stacks across four distinct RAG pipelines. They evaluated performance using frameworks like Auto-ARGUE and MiRAGE. The core finding is a strong correlation between coverage-based retrieval metrics and the 'nugget coverage'—the completeness of key information—in the generated responses. This relationship holds at both the topic and system levels, though it is strongest when the retrieval objectives are closely aligned with the generation goals. The research also notes that more complex, iterative RAG pipelines can partially decouple final answer quality from initial retrieval effectiveness.
- Study tested 15 text and 10 multimodal retrieval systems across three major RAG benchmarks (TREC NeuCLIR, TREC RAG, WikiVideo).
- Found strong correlation between coverage-based retrieval metrics and information 'nugget coverage' in final AI-generated answers.
- Provides empirical evidence that retrieval quality can serve as a reliable early indicator for RAG system performance.
Why It Matters
This gives developers a concrete, measurable way to predict and improve their RAG systems' output quality before full generation, saving time and resources.