New GPU method speeds LLM inference 1.64x with moderate sparsity

Pruning large language models (LLMs) to introduce sparsity in weight matrices can reduce computational cost, but existing GPU sparse matrix multiplication (SpMM) kernels fail to beat dense counterparts at moderate sparsity levels (~50%). To solve this, researchers from Zhejiang University (Tao Lu et al.) propose a new GPU inference method designed specifically for this sparsity range. They introduce a three-layer matrix storage format: (i) a Sparse-TC layer that enables sparse tensor cores to accelerate SpMM; (ii) a Slot-Filling layer that compresses the matrix using parallel differential distance, enabling low-cost on-chip decoding; and (iii) a lightweight Residual Layer to ensure correct SpMM computation. Their custom SpMM kernel jointly utilizes sparse tensor cores and CUDA cores, overlapping on-chip computation with memory access for an efficient execution pipeline.

Evaluated on modern GPUs with high-bandwidth memory (HBM), the method is the first to outperform dense matrix multiplication at moderate sparsity. It achieves up to 1.64x kernel-level speedup over SpInfer (EuroSys'25, Best Paper) and up to 1.41x end-to-end speedups over FlashLLM (VLDB'24). While the focus is on 50% sparsity, the approach could extend to other levels. The open-source code is available on GitHub. This breakthrough makes LLM inference cheaper and faster without sacrificing model quality, a critical advancement for deploying large models in production.