SproutRAG boosts long-document RAG by 6.1% with attention-guided tree search
No extra LLM calls needed: new hierarchical RAG framework learns semantic structure from attention heads.
Retrieval-augmented generation (RAG) systems face a fundamental trade-off: finer retrieval chunks capture precise facts but lose context, while larger chunks maintain coherence but dilute relevance. Existing solutions rely on expensive LLM calls for chunking, fixed context expansion, or hierarchical summarization—each introducing latency, cost, or information loss.
SproutRAG, developed by researchers at the University of British Columbia and ServiceNow, solves this with an attention-guided hierarchical framework. It first encodes sentence-level embeddings, then uses learned attention heads and layers to progressively merge semantically related sentences into larger chunks, forming a binary tree. At retrieval time, hierarchical beam search explores multiple granularities simultaneously, selecting the most coherent multi-sentence candidates—no extra LLM inference required. The entire system is trained end-to-end: the embedding model and tree structure jointly optimize retrieval quality. On four benchmarks spanning scientific papers, legal documents, and open-domain QA, SproutRAG achieves a 6.1% average improvement in information efficiency (IE) over the strongest baseline, with code publicly available.
- Uses learned inter-sentence attention to build a binary chunking tree, avoiding costly LLM calls during indexing and retrieval.
- Hierarchical beam search retrieves candidates at multiple granularities, improving multi-sentence relevance beyond flat retrieval.
- 6.1% average improvement in information efficiency across four benchmarks (scientific, legal, open-domain).
Why It Matters
SproutRAG makes long-document RAG faster and cheaper by eliminating extra LLM calls while preserving contextual coherence.