New AI Breakthrough Cuts Chatbot Memory Costs 8x

The key-value cache is the biggest memory bottleneck in long-context large language model inference. Researchers found that uniform quantization hits a “quality cliff”—a narrow range where performance suddenly collapses. Above that cliff, an importance-aware mixed-precision approach called SemKV ranks tokens and assigns two adjacent above-cliff precisions, cutting KV cache storage by 6.0x with no statistically detectable quality difference from full precision, and outperforming token pruning even when pruning gets 1.5x more memory. Replacing the base quantizer with a distortion-optimized one pushes the no-detectable-loss operating point to 7.9x. The recipe: measure the cliff for your deployment, then interpolate above it.