OpRAG runtime boosts GPU RAG pipelines by 20%, cuts latency 59%

OpRAG is a new distributed runtime from Arup Kumar Sarker and colleagues at the University of Virginia and Rutgers that targets the orchestration bottleneck in agentic retrieval-augmented generation (RAG). While LLM decoding is GPU-bound, the surrounding pipeline—preprocessing, embedding, retrieval, memory access, context construction, and index updates—often stalls on serialization, fragmented scheduling, and poor CPU-GPU overlap. OpRAG models these stages as resource-aware operators and compiles them into communication-aware execution graphs, combining an Arrow zero-copy data plane, persistent workers, bounded queues, CPU tokenizer prefetching, batched GPU embedding, and overlapped retrieval/generation execution. The result is substantial speedup without modifying the LLM kernel.

In end-to-end GPU pipeline tests with Llama3-8B and Mistral-7B (FlashAttention-2, BF16, 32K RAG chunks), OpRAG improved throughput by 16.16% and 15.66% over the nearest competitor, and by 20.57% and 20.71% over RayScalableRAG. Against popular frameworks LangChain, LangGraph, CrewAI, and AutoGen, it was 17.77% and 17.48% faster than the best baseline. In Hybird-style query serving, OpRAG reduced hybrid retrieval latency by 59.2–59.6% and generation-scenario latency by 52.5–53.6%, while preserving perfect Recall@5. The authors show that optimizing distributed orchestration delivers massive gains for GPU-backed multi-stage RAG—no decoder changes needed.