RAGPerf: An End-to-End Benchmarking Framework for Retrieval-Augmented Generation Systems

A team of researchers, including Shaobo Li and seven others, has introduced RAGPerf, a comprehensive open-source framework designed to benchmark the performance of Retrieval-Augmented Generation (RAG) systems. RAGPerf addresses a critical gap in the AI development landscape by providing a standardized way to measure the complex, multi-stage pipelines that combine document retrieval with large language model generation. The framework's key innovation is its modular design, which decouples the RAG workflow into distinct components—embedding, indexing, retrieval, reranking, and generation—allowing for detailed profiling and fine-tuning of each stage.

RAGPerf is highly configurable, enabling users to test with different embedding models, major vector databases like LanceDB, Milvus, and Qdrant, and various LLMs for generation. It includes a workload generator that models real-world scenarios using diverse data types (text, PDF, code, audio) and query patterns. The framework automates the collection of both performance metrics, such as query throughput and GPU memory footprint, and accuracy metrics, including context recall and factual consistency. The team's evaluation shows the tool adds negligible overhead, making it a practical choice for developers.

By open-sourcing the codebase, the researchers aim to provide the community with a vital tool for making informed architectural decisions. As RAG systems become central to enterprise AI applications, from chatbots to internal knowledge bases, RAGPerf offers a data-driven method to compare components, identify bottlenecks, and balance speed against accuracy. This moves RAG development from a trial-and-error process toward a more systematic engineering discipline.