New adaptive KV caching speeds up LLM serving by 12.6%

Key-value (KV) caching is critical for efficient LLM inference, but standard least-recently-used (LRU) eviction policies cause cache thrashing when multiple diverse workloads share the same cache. A team of researchers from Johns Hopkins University and Argonne National Laboratory (Yang Shen, Meghana Madhyastha, Robert Underwood, Bogdan Nicolae, Randal Burns) has developed a new approach that dynamically balances recency and frequency of KV blocks. Their method, presented at the ICML 2026 Workshop on Resource-Adaptive Foundation Model Inference, adaptively allocates cache space to both recently accessed and frequently accessed blocks, preventing unrelated workloads from flushing each other's caches.

In evaluations using vLLM as the baseline, the adaptive caching achieved a KV cache hit rate improvement of up to 10.8% on synthetic document question-answering workloads, reducing time-to-first-token by 12.6%—a significant latency win for interactive applications. On real-world conversation workloads, gains were more modest at 2.1% hit rate and 2.0% latency improvement, but the method proved robust across batch inference settings and offered clear interpretability. The paper is available on arXiv and accepted at the AdaptFM workshop, highlighting a practical step toward smarter memory management in LLM serving infrastructure.