MosaicKV compresses KV cache 3x with 16x speedup for long-context LLMs
A new technique cuts memory 3x while boosting throughput 7.3x—with minimal accuracy loss.
MosaicKV tackles the memory bottleneck of long-context LLMs, where the key-value (KV) cache grows linearly with input length, often exhausting GPU memory. Previous compression methods target only one dimension—sequence or channel—leaving limited headroom. MosaicKV applies dynamic two-dimensional compression, identifying important elements per KV vector and selecting strategies at the segment granularity. This avoids the accuracy loss from naive two-dimensional compression.
Evaluated on an H800 GPU with multiple LLMs, MosaicKV delivers 16x attention speedup, 4.8x lower decode latency, 7.3x higher throughput, and 3x memory reduction versus uncompressed baselines. Accuracy drops only 1.76% on average across LongBench and RULER. The system uses underutilized GPU and CPU resources to manage compressed caches and accelerate attention, making it practical for real-world deployment.
- 16x attention speedup and 4.8x lower decode latency on H800 GPU
- 3x memory reduction with only 1.76% average accuracy loss
- Dynamic two-dimensional compression at KV segment granularity avoids global pattern issues
Why It Matters
MosaicKV enables cost-effective long-context LLM serving, making million-token contexts practical for production systems.