New 1.125-Bit Quantization Method Keeps LLM Accuracy Near FP16

Traditional layer-wise post-training quantization of large language models minimizes reconstruction error per layer in isolation, allowing errors to accumulate across depth—especially severe at extreme low-bit levels. Ryona Noda's paper introduces two complementary mechanisms to address this. First, cross-layer error compensation formulates quantization as a joint optimization across all layers, recursively tracking accumulated error via e_{l+1} = A_l e_l + q_l, where A_l is a propagation operator derived from the layer's input differential. This makes the recursion exact for arbitrary nonlinear layers, enabling a forward-difference implementation. Binary weights are optimized using mirror-descent with annealed inverse temperature.

The second mechanism, finite-sample feature-statistics matching, aligns means, projected covariances, and centered empirical kernels between full-precision and quantized networks under relative normalization. On Qwen2.5-1.5B at 1.125-bit group-binary weights, error compensation alone yields a perplexity ratio of 9.56 ± 0.15 over FP16, beating logit distillation (14.09) by 32% relative (more than 8 sigma). The same objective transfers to 4-bit quantization (1.060 vs 1.088 for layer-local). Out-of-domain evaluations on C4 and CNN/DailyMail show error compensation's advantage grows off-domain, while statistics matching keeps feature discrepancy low (0.42-0.88 vs 1.41-2.99 without it), demonstrating a clear division of labor.