AgentKVShift enables 3.5x faster KV cache reuse for AI agents

Memory-augmented LLM agents maintain context across hundreds of interactions by curating retrieved content with metadata like summaries and tags. A major bottleneck is that every retrieval triggers full re-encoding of structured memory units into Key-Value (KV) states, dominating prefill latency. Existing training-free KV reuse methods were designed for raw RAG passages, not the structured memories used by agents, leading to performance degradation. AgentKVShift, a new paper from researchers at UC San Diego and collaborators, solves this with a probe-guided KV residual correction method. It exploits the insight that per-memory KV reuse residual decomposes into a shared offset plus small token-wise fluctuations. By estimating this offset from a small probe set, AgentKVShift corrects every reused token with a single weighted correction — turning the refresh budget into useful signal across the entire chunk.

Evaluated across four open-source LLMs from 3B to 32B parameters on long-horizon dialogue and agentic benchmarks, AgentKVShift achieves near full recompute performance while refreshing only 10-30% of the cache. This requires up to 5x less recompute than prior methods, which only reach similar quality at 45-55% refresh. In this regime, it delivers 2-3.5x prefill speedups over no-KV-reuse on a single A100. Additionally, AgentKVShift orthogonally composes with KV cache quantization, retaining over 2x the F1 score under aggressive 2- and 4-bit settings compared to prior reuse methods. This makes it a practical, drop-in optimization for building faster, cheaper long-running AI agents.