CA-RAG cuts token costs 26% by routing queries per retrieval depth

Retrieval-augmented generation (RAG) faces a fundamental tradeoff: deeper retrieval improves factual grounding but increases token costs and latency. Static configurations fail across heterogeneous query workloads—simple definitional queries waste budget, while complex analytical prompts need deeper retrieval. Sanjay Mishra's paper introduces Cost-Aware RAG (CA-RAG), a per-query routing framework that selects from a discrete catalog of strategy bundles, each coupling a retrieval depth (from retrieval-free direct inference to top-k=10 dense retrieval) with a fixed generation profile. The router maximizes a scalar utility combining an estimated quality prior with normalized penalties for predicted latency and total billed tokens.

Implemented with FAISS-backed dense retrieval and OpenAI chat/embedding APIs, CA-RAG was evaluated on a 28-query benchmark spanning four bundles. Results show the router dynamically exercises all bundles, achieving 26% fewer billed tokens than always-heavy retrieval and 34% lower mean latency than always-direct inference while maintaining equivalent answer quality. Per-query delta analysis reveals savings are non-uniform and concentrated in simpler queries, motivating complexity-aware guardrails. Sensitivity analysis confirms the same bundle catalog supports multiple cost-latency-quality operating points through weight adjustment alone. All results are fully reproducible from logged CSV artifacts, providing a transparent foundation for cost-conscious LLM deployments.