New Research Makes AI Answer-Bots Cheaper and More Accurate

RAG systems typically retrieve a fixed number of documents (top-k) to ground generation — but that static approach is brittle: simple queries get over-retrieved, adding noise and cost, while complex queries get under-retrieved, causing recall failures that cascade into incorrect answers. A new paper proposes a practical, general framework for query-adaptive retrieval depth. Offline, it estimates per-query retrieval difficulty by measuring NDCG under the default retriever and derives a query-specific "saturation" point k* from the NDCG-k curve. Because computing those signals online is expensive, it clusters a large set of queries in embedding space and summarizes each cluster with a recommended retrieval depth targeting high coverage (e.g., around 95%) using a mean-plus-variance rule. At runtime, the system assigns an incoming query to a cluster and selects the corresponding top-k in constant time. Compared with post-retrieval confidence methods that rely on clustering retrieved documents, this approach is pre-retrieval and query-centric, making it robust in heterogeneous, case-like corpora and applicable across domains such as legal, healthcare, finance, and enterprise search. The framework has been tested in full-traffic queries, improving F1 by over 36% while reducing token usage by 14% on low-complexity clusters without accuracy loss. The paper was accepted to the Applied Research Track of CIKM 2026.