Huawei's KVarN KV-cache quant achieves q5 quality at 4-bit in llama.cpp
3–5× compression with better precision than TurboQuant – and it's Apache 2.0.
A developer implemented Huawei's KVarN KV-cache quantization method in their own llama.cpp fork, BeeLlama.cpp v0.3.2 Preview. KVarN, published under Apache 2.0, promises 3–5× compression without the slowdowns seen in TurboQuant, and maintains reasoning quality. The implementation currently supports Qwen 3.6 27B and Gemma 4 31B models, with a single --cache-type-kvarn4 flag to enable. The developer ran Kullback-Leibler divergence (KLD) benchmarks on 50+ quantization pairs to assess real-world precision, rejecting simpler perplexity metrics.
The results are impressive: kvarn4-kvarn4 uses only 27.9% of the original cache size yet delivers mean KLD of 0.002974—better than q5_0 (34.4% size, KLD 0.003206) and significantly better than TurboQuant's q5_0-turbo3_tcq (27.3% size, KLD 0.005471). At 99.9% KLD, kvarn4-kvarn4 (0.094819) nearly matches q5_0 (0.099073) and beats TurboQuant. The developer claims q5 quality at 4-bit and q4 quality at 3.5-bit. Throughput (760.88 tok/s) is lower than some quants, but the implementation is raw; the original paper reported speed gains. For VRAM-constrained users, this is a significant step forward in running large contexts on consumer hardware.
- KVarN provides 3–5× KV-cache compression with Apache 2.0 license, now available in BeeLlama.cpp v0.3.2 Preview.
- Benchmarks show kvarn4-kvarn4 at 27.9% cache size achieves mean KLD of 0.002974, outperforming q5_0 (0.003206) and TurboQuant (0.005471).
- Supports Qwen 3.6 27B and Gemma 4 31B; claims q5 quality at 4-bit and q4 quality at 3.5-bit.
Why It Matters
Enables larger context windows on consumer GPUs with minimal precision loss, democratizing long-context LLM inference.