OasisKV boosts LLM throughput 2.1x by offloading KV cache from HBM

As LLMs shift toward long-context and reasoning-heavy tasks, inference is increasingly bottlenecked by memory rather than compute. The key-value (KV) cache—which stores attention state for every token—now dominates HBM capacity and traffic, limiting batch size and system throughput. OasisKV, a new system from researchers including Can Xiao and Sukmin Cho, tackles this head-on by decoupling full KV storage from HBM during decode. Instead of keeping the entire cache in scarce high-bandwidth memory, OasisKV stores it in cheaper host or remote memory tiers and only loads what the attention mechanism actually needs.

The key innovation is predicting which KV entries will be important before the next decode step. OasisKV leverages the lookahead tokens drafted by speculative decoding (SD) to forecast future attention patterns with high accuracy. A background pipeline then identifies and prefetches the relevant KV blocks, staging them in HBM just in time. Using a 2,048-token KV budget, the system stays within 0.7 accuracy points of full attention, proving that sparse attention can be both practical and lossless in practice.

Built on vLLM, OasisKV turns sparsity into real throughput gains: 1.69x over dense vLLM on reasoning workloads with only 0.1 points of accuracy loss, and up to 2.1x on multi-GPU long-context serving. In a prefill-decode disaggregated setup, it achieves roughly 2x dense throughput while admitting each request with 6.5–9.7x less KV and holding 2.2–2.6x less decode-node host memory than full KV transfer. This makes OasisKV a promising direction for scaling LLM inference on existing hardware, especially for deployment scenarios where HBM is a costly bottleneck.