New study reveals optimal KV-cache compression methods for LLMs

A new arXiv paper by Paolo D'Alberto and colleagues provides a rigorous ablation and statistical validation study comparing KV-cache compression methods critical for efficient LLM inference. The work systematically compares Turbo-Quant and SpectralQuant codecs, along with non-dominated schemes like Walsh-Hadamard Transform (WHT) rotation combined with Beta Lloyd-Max quantization and quantized Johnson-Lindenstrauss (QJL) transforms. The authors employ a careful statistical inference methodology that separates systematic codec differences from implementation variance, allowing for reliable head-to-head performance assessment.

The research reveals that eigenbasis-based methods, while powerful in structured data regimes, fail on heavy-tailed data distributions due to covariance instability. Conversely, methods like WHT rotation with Beta Lloyd-Max show more robust performance across diverse data types. A particularly notable finding is that the effective semantic dimension adapts to calibration budgets rather than reflecting the true underlying data rank—a nuance that significantly impacts deployment decisions. These insights guide practitioners in selecting the optimal KV-cache compression strategy based on their specific data characteristics, with implications for reducing memory footprint and latency in production LLM systems.