New quantization method finds low-loss subspaces to preserve model accuracy

Quantizing neural networks (reducing parameter precision from 32-bit floats to e.g. 8-bit integers) is critical for deploying models on edge devices, but discrete constraints perturb parameters away from the carefully optimized minimum, degrading accuracy. Traditional quantization-aware training (QAT) mitigates this by simulating quantization during training, often using the straight-through estimator (STE) — a heuristic that approximates gradients through discrete operations. However, STE can introduce bias and instability.

Protsenko and colleagues propose a radically different approach: instead of forcing parameters into a discrete set during training, they learn quantization-friendly subspaces in the continuous weight space. They exploit recent findings that low-loss full-precision solutions reside in connected low-loss subspaces. By learning linear paths (effectively one-dimensional subspaces) that minimize loss, they ensure the midpoint of each path lies within a region where quantizing the weights causes minimal performance drop. The result is a quantized model that retains accuracy comparable to QAT, but without any STE or training-time discretization. The method is demonstrated on image classification tasks (CVPR domain) and could simplify the quantization pipeline significantly.