This New AI Memory System Slashes LLM Bottlenecks by 9.7x — But Only for One Type of Model
Sparse attention models finally get efficient KV cache with CXL's fine-grained fetch.
A new paper from researchers (including Ruiyang Ma, Teng Ma, and others) introduces SAC, a disaggregated KV cache system built specifically for sparse attention large language models. As LLMs scale to longer contexts, the memory needed for the key-value (KV) cache becomes the primary bottleneck. Traditional RDMA-based disaggregated memory pools fetch the entire prefix KV cache from remote storage to local memory before decoding, which is wasteful for sparse attention models that only activate a small fraction of KV entries during inference.
SAC leverages Compute Express Link (CXL), a low-latency interconnect supporting cache-line granularity load/store operations. This allows on-demand fetching of only the required top-k KV entries, eliminating transmission bottlenecks and local memory waste. Evaluated on DeepSeek-V3.2 using SGLang, SAC delivers 2.1x higher throughput, 9.7x lower time-to-first-token (TTFT), and 1.8x lower time-between-tokens (TBT) versus RDMA baselines. The results position CXL-based disaggregation as the superior infrastructure for emerging sparse attention models, promising faster and more memory-efficient long-context LLM inference.
- SAC uses CXL's cache-line granularity to fetch only active top-k KV entries, not entire prefixes.
- On DeepSeek-V3.2, SAC achieves 2.1x higher throughput, 9.7x lower TTFT, and 1.8x lower TBT vs RDMA.
- First KV cache system optimized for sparse attention models, solving memory and transmission bottlenecks.
Why It Matters
CXL disaggregation makes long-context LLM inference affordable by slashing latency and bandwidth waste.