PEEK: KV Cache Management Cuts LLM Serving Latency 7.9x
New algorithm boosts cache hits 3x and time-to-first-token 7.9x on H100 clusters.
PEEK tackles a critical bottleneck in LLM inference: managing the ever-growing KV cache to maximize reuse while minimizing latency. Traditional systems evict cache blocks reactively, missing the opportunity to prefetch for future requests. PEEK avoids this by maintaining an incremental radix tree over the pending request queue, surfacing prefix-sharing clusters that no existing engine exposes. A low-overhead dual-walk algorithm matches the tree against the engine’s current prefix cache to find the longest prefix for every waiting request. PEEK then admits cluster pioneers first, so sibling requests inherit the freshly cached prefix. A co-designed eviction hook protects cache blocks that are ancestral to queued demand, and a multi-lane stride scheduler bounds starvation.
On real hardware with SGLang and vLLM across five diverse workloads, PEEK delivers up to 3.0×/2.6× cache hit ratio improvement, 7.9×/7.1× faster time-to-first-token (TTFT), 6.7×/5.5× end-to-end latency reduction, and 3.6×/4.5× throughput gains over each engine’s strongest stock baseline. The wins persist as KV-cache pressure and inference parallelism scale. For workloads without exploitable prefix structure, PEEK matches baselines within noise. The paper provides 26 pages of detailed analysis, making it a promising contribution to efficient LLM serving infrastructure.
- Uses an incremental radix tree over the pending queue to expose prefix-sharing clusters not surfaced by existing engines.
- Achieves up to 7.9× time-to-first-token (TTFT) improvement on SGLang and 7.1× on vLLM across H100 clusters.
- Co-designed eviction hook protects cache blocks ancestral to queued demand, preventing premature eviction.
Why It Matters
PEEK enables dramatic latency and throughput improvements for LLM serving without hardware upgrades, critical for cost-efficient AI inference at scale.