DeepSeek V4 Flash KV quantization degrades quality; Qwen 397B handles it

A Reddit user (erazortt) shared benchmark results strongly advising against quantizing the KV cache (key-value cache, the memory that stores attention computations) on DeepSeek V4 Flash (DS4F). The tests compare BF16 to Q8 quantization across perplexity (PPL), Kullback-Leibler divergence (KLD), and token probability shifts. For DS4F, mean PPL increased from 5.8397 to 5.8771 (a 0.64% relative rise), while mean KLD was 0.1459—with a maximum KLD of 12.47 and 95% of sequences exceeding 0.65. Token probabilities shifted dramatically: mean absolute Δp was 0.007% but RMS Δp reached 11.88%, and 5% of tokens had probability drops over 14.5%. Only 87.2% of responses kept the same top-p token after quantization.

In contrast, erazortt benchmarks Qwen 397B under the same conditions. Qwen showed near-imperceptible degradation: mean PPL rose just 0.03%, mean KLD was 0.0036, the maximum KLD was 2.22, and 95% of sequences stayed below 0.014 KLD. RMS Δp was only 2.02%, and same-top-p agreement was 97.9%. The implication is clear: KV cache quantization is architecture-specific. DeepSeek V4 Flash's attention mechanism likely relies on precise key/value precision, making aggressive 8-bit quantization (Q8) risky for production use. The author recommends keeping BF16 KV caches for DS4F unless memory constraints are extreme, and warns that generic quantization guidance from other models (like Qwen) doesn't transfer to DS4F.