SLQ quantizes LLMs to 3.3 bits with zero accuracy loss

A new paper from IST Austria introduces SLQ (Statistically-Lossless Quantization), a method that compresses large language models without losing fidelity—going beyond traditional lossy approaches like GPTQ and AWQ. The key innovation is defining three levels of losslessness: task-lossless (preserving zero-shot benchmark accuracy within natural variance), distribution-lossless (next-token distribution indistinguishable from original, measured by Expected Acceptance Rate ≥ 0.99), and a gamma-squared variance law proving asymmetric quantization is necessary for distribution-lossless fidelity.

SLQ uses layer-wise non-uniform asymmetric quantization with a wide bitwidth search. Results show task-lossless compression at 3.3-4 bits per parameter and distribution-lossless at 5-6 bits, with inference speedups of 1.7-3.6x over FP16 on GPUs and CPUs (including llama.cpp support). This bridges the gap between aggressive lossy compression and slower lossless methods, making it practical for production deployment of large models while guaranteeing reliability.