Bicache boosts DLM throughput by up to 98% without accuracy loss
Existing KV caching breaks diffusion models; new technique fixes it.
Standard key-value (KV) caching for shared prefixes is crucial for efficient LLM serving. But diffusion language models (DLMs)—which use bidirectional attention—break this approach: updating any token alters the entire context’s KVs, so caching previously computed KVs corrupts the shared prefix and collapses model accuracy to near zero.
Researchers now present bicache, the first prefix-caching method designed for DLMs. It exploits a key insight: shared prefix KVs remain stable in shallow layers, while the safe depth depends on the fraction of prefix tokens in each request. Bicache dynamically identifies that depth per request, reusing prefix KVs only where safe and skipping redundant computation elsewhere. Evaluations show bicache boosts serving throughput by 36.3% to 98.3% compared to prior techniques, with accuracy loss limited to 0–1.8%—making high-throughput DLM serving practical for the first time.
- Bicache reuses shared prefix KVs only in shallow layers, avoiding accuracy collapse seen with standard caching in DLMs.
- Dynamic per-request depth selection based on prefix token ratio eliminates redundant computation.
- Serving throughput improved 36.3%–98.3% over existing methods; accuracy loss ≤1.8%.
Why It Matters
Unlocks real-time, high-throughput serving for diffusion models, a key step for interactive AI applications.