New Trick Shrinks AI Models So They Run on Cheaper Hardware

PRQuant (Permutation Residual Quantization) is a training-free, low-overhead framework that combines channel reorganization with static weight-side residual compensation. Accuracy in low-bit quantization of linear layers is often dominated by a small number of outliers; existing methods like smoothing, rotation, or residual-based approaches can mitigate this but often introduce new accuracy bottlenecks to weights, and most are implemented as online approaches that can mean heavy execution overheads. After AWQ-style scaling, PRQuant identifies the input channels that contribute most to weight quantization error, permutes them into contiguous tail blocks, and constructs their residual weight sub-tensors offline. During inference, that contiguous structure lets the activation side use tail blocks without an expensive online gathering operation and turns scattered residual compensation into a regular tail-augmented GEMM, substantially reducing latency. Experiments show PRQuant effectively reduces down-projection reconstruction error, and ablation studies confirm smoothing and residual compensation are the primary drivers of numerical improvement, while permutation provides a consistent marginal numerical benefit and, more importantly, enables a hardware-friendly contiguous layout that eliminates dynamic gathering overhead. Overall, it outperforms default MXFP4 and the evaluated PTQ baselines in average accuracy across five downstream benchmarks, improving over MXFP4 by 1.24 on Qwen3-4B-Instruct-2507 and 0.55 on Qwen3-30B-A3B-Instruct-2507.