SVD-RAG: New technique makes RAG 317x faster, 85% cheaper
No more expensive LLM summarization—SVD-RAG builds retrieval trees in 0.1 seconds.
Retrieval-Augmented Generation (RAG) systems typically rely on hierarchical document structures to improve search accuracy, but methods like RAPTOR require expensive LLM-based abstractive summarization at every internal tree node—making large-scale deployment costly. SVD-RAG, developed by Zhihui Sun, solves this by applying Singular Value Decomposition (SVD) to dense sentence embedding matrices for extractive summarization. Unlike classical LSA that uses sparse TF-IDF matrices, SVD-RAG leverages modern embedding models to identify the most informative sentences via their energy contribution in principal components.
In controlled head-to-head tests on identical corpora, SVD-RAG achieved retrieval quality within 1-5% of RAPTOR (MRR 0.867 vs 0.875, Recall@1 0.483 vs 0.458) while building the tree 317x faster (0.1s vs 31.7s) and cutting token consumption by ~85%. On a scaled multi-topic benchmark with 205 chunks and 100 queries, SVD-RAG delivered a 4.2x improvement in Recall@1 and 3.1x improvement in MRR over flat embedding retrieval. The approach is deterministic, cost-efficient (no API calls beyond initial embedding), and content-adaptive via an energy-ratio threshold. Released as an open-source Python package, SVD-RAG makes hierarchical RAG practical for enterprise knowledge systems at scale.
- SVD-RAG replaces expensive LLM-based summarization with SVD extractive summarization, cutting token usage by ~85%.
- Tree construction is 317x faster than RAPTOR (0.1s vs 31.7s) while maintaining retrieval quality within 1-5% (MRR 0.867 vs 0.875).
- Achieves a 4.2x improvement in Recall@1 over flat retrieval on multi-topic benchmarks and is released as an open-source Python package.
Why It Matters
Makes hierarchical RAG practical at scale, drastically lowering cost and latency for enterprise knowledge retrieval systems.