KV-cache optimization benchmark: SnapKV best throughput, KIVI4 stable quality
Compression ratio alone misleads; workload-aware selection is key for long-context LLM serving.
A new study from researchers Nikita Agrawal and Ruben Mayer provides the first systematic benchmark of KV-cache optimization techniques for long-context LLM serving. The paper evaluates four representative mechanisms—KIVI, TurboQuant, SnapKV, and CaM—spanning quantization, pruning, and merging approaches. Tests were conducted on Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3 using LongBench-style workloads: multi-document QA, single-document QA, few-shot learning, and summarization.
Key results reveal that compression ratio alone is a poor predictor of end-to-end system performance. KIVI4 provides the most stable task quality across models, SnapKV achieves the strongest long-context throughput, and CaM yields large gains on selected QA workloads but exhibits substantial sensitivity in both quality and realized compression ratio. The findings motivate workload-aware selection of KV-cache mechanisms rather than a one-size-fits-all approach, offering practical deployment guidance for systems serving long-context language models.
- SnapKV delivers the strongest long-context throughput on Llama-3.1-8B and Mistral-7B models.
- KIVI4 offers the most stable quality across tasks and models, while compression ratio alone is a poor performance predictor.
- CaM provides large gains on specific QA workloads but is highly sensitive to workload changes.
Why It Matters
Guides engineers in selecting KV-cache optimizations for efficient, workload-aware long-context LLM deployment.