Multi-Scale Dequant: New Method Eliminates Dequantization Bottleneck for LLM Inference

A new paper from Lingchao Zheng and colleagues introduces Multi-Scale Dequant (MSD), a quantization framework that tackles the dequantization bottleneck in LLM inference. On modern AI accelerators with decoupled compute units (e.g., Ascend NPUs), dequantization—converting low-bit weights back to high precision—can consume more cycles than matrix multiplication itself, leaving tensor cores underutilized. MSD eliminates this by decomposing high-precision BF16 activations into multiple low-precision components, each multiplied directly with quantized weights via native hardware-accelerated GEMM. This shifts the paradigm from precision conversion to multi-scale approximation. For INT8 weights, two-pass INT8 decomposition achieves near 16 effective bits; for MXFP4 weights, two-pass decomposition yields ~6.6 effective bits with error bound 1/64 per block, outperforming single-pass MXFP8 while maintaining the same GEMM compute time.

MSD also addresses memory bottlenecks in attention mechanisms. The authors derive closed-form latency and HBM traffic models showing that MSD avoids Vector-Cube pipeline stalls caused by dequantization, reducing KV cache HBM traffic by up to 2.5x. Numerical simulations on matrix multiplication and Flash Attention kernels confirm that MSD does not degrade accuracy compared to standard dequantization baselines—in many settings it achieves lower L2 error. This work is particularly relevant for hardware like Ascend NPUs, where dequantization is a known bottleneck, but the principle applies to any architecture with decoupled compute. By removing the precision conversion step from the critical path, MSD offers a practical path to faster, more memory-efficient LLM inference without sacrificing model quality.