DeepSeek Sparse Attention slashes long-context LLM costs by 80%
Fixed top-2048 tokens per query cuts compute 98% at 128K context
DeepSeek Sparse Attention (DSA) breaks the quadratic scaling of dense attention by attending over only the top-2048 most relevant tokens per query, regardless of context length. At 128K context, this cuts attention compute by ~98% and reduces per-user KV memory from ~40 GB to just ~0.6 GB, enabling far higher GPU concurrency. DSA uses a two-stage mechanism: first, a lightweight "lightning indexer" computes low-rank query-key projections to score token blocks; second, only the top-K blocks are loaded into compute units. This learned sparsity pattern, built into models like GLM-5.1 and DeepSeek V4, avoids the catastrophic retrieval failures of sliding-window attention while matching dense attention on long-range benchmarks.
DSA complements existing techniques: it reduces both FLOPs and memory bandwidth, unlike FlashAttention which only optimizes memory I/O. At 128K-512K contexts, DSA matches Ring Attention's memory savings without inter-GPU communication overhead. For contexts over 1M tokens, DSA can be combined with Ring Attention for further per-rank FLOP savings. The deployment stack includes support for vLLM and SGLang, with benchmarks showing ~98-99% retrieval accuracy on RULER. This makes DSA a practical, cost-efficient solution for serving long-context LLMs on existing GPU hardware.
- DSA reduces attention compute by ~98% at 128K context by selecting a fixed top-2048 tokens per query
- KV memory per user drops from ~40 GB (dense) to ~0.6 GB (DSA), enabling higher concurrency
- Learned sparsity avoids catastrophic failure of sliding-window on long-range retrieval tasks, achieving ~98-99% accuracy on RULER
Why It Matters
Reduces GPU cloud costs for long-context LLMs, making multi-million-token models practical on existing hardware.