Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression

Researchers Longsheng Zhou and Yu Shen have published a paper titled 'Prune-Quantize-Distill: An Ordered Pipeline for Efficient Neural Network Compression,' addressing a critical gap in AI deployment. The study highlights that common compression proxies like parameter count often fail to predict actual wall-clock inference time, especially on standard CPUs where unstructured sparsity can even slow down execution due to irregular memory access. Their solution is a meticulously ordered three-stage pipeline that first applies unstructured pruning to reduce model capacity, then implements INT8 quantization-aware training (QAT) for the dominant runtime benefit, and finally uses knowledge distillation (KD) to recover accuracy within the already compressed model.

Empirical evaluation on CIFAR-10 and CIFAR-100 datasets using ResNet-18, WRN-28-10, and VGG-16-BN backbones proved the pipeline's effectiveness. The ordered approach consistently achieved a stronger frontier in the joint accuracy-size-latency space than applying techniques in isolation or in a different sequence. A key finding is that INT8 QAT provides the most significant speed-up, while pruning primarily acts as a pre-conditioner that makes the model more robust to subsequent low-precision optimization. The final distillation step fine-tunes accuracy without altering the deployment-ready, sparse INT8 format. Controlled ablations with a fixed 20/40/40 epoch allocation confirmed that the proposed order generally performs best among all tested permutations, making it a simple yet powerful guideline for practitioners.