Bojie Li's KV Cache Breakthrough: Editable and Composable for 14.9x Speedup

Current prefix caching in LLMs is brittle: changing one field invalidates the entire downstream cache. Bojie Li's new paper reveals that KV cache can be treated as a notebook of memoized conclusions, making it both editable and composable. By overwriting only the key/value vectors of the changed field, the model can reuse the rest of the cache—but surprisingly, the field's own KV drives under 1% of the decision. The real influence comes from downstream 'notes' written during prefill. This means that with chain-of-thought, editing just the field recovers the correct decision at only ~1% of compute cost, without full recomputation.

Beyond editing, the KV cache is position-portable. Precompiled skills (e.g., reasoning patterns) can be RoPE-repositioned and spliced into any new context, achieving logit cosine similarity of 0.90–0.999 with full recompute across twelve models—while reducing time-to-first-token from O(L²) to O(L). A unified edit+compose agent stays decision-identical to recompute at up to 14.9x lower latency. The approach works across scales, quantization, Mixture-of-Experts, and multimodal caches, and extends to various attention variants via small adapters. Because edits are append-only, it seamlessly integrates with production prefix caching: in an online vLLM benchmark, it maintains a 98.5% cache hit rate and cuts p90 time-to-first-token by 53–398x.