New arXiv Paper Proposes Buying KV Caches to Slash AI Compute 50x

A new arXiv paper titled "Can I Buy Your KV Cache?" by Luoyuan Zhang proposes a deceptively simple yet radical idea: instead of every AI agent recomputing the key-value (KV) cache from scratch for the same document, let a publisher precompute it once and sell access to agents. The paper demonstrates that loading a precomputed KV cache and continuing inference is token-exact—matching prefill on 24/24 greedy tokens and at the logits level—with zero accuracy loss. On Qwen3-4B, reusing a precomputed KV cache is 9-50x cheaper in compute than full prefill, and the savings grow superlinearly with document length because prefill attention scales as L².

The paper quantifies the massive economic opportunity: serving a single hot 3774-token document to 80 million agents would cost ~$1.5M in recompute but only ~$0.03M in reuse compute—a 49.7x reduction. However, the catch is egress: shipping KV caches is nearly incompressible, so per-load egress costs can exceed the prefill savings. The solution is provider-side hosting, exactly how production prompt caching works today. The paper frames this as an "agent-native prefill CDN" and identifies lossless KV compression and cross-party payment layers as open problems. This could fundamentally reshape AI inference economics by turning compute into a tradeable commodity.