Compound sparsity combines static and dynamic pruning to push LLM compression limits

Large language models are typically compressed either through static parameter pruning or dynamic token-level computation, but aggressive sparsification often leads to rapid performance degradation beyond a certain boundary. A new paper from researchers Chao Han, Haozhe Hu, and Xiaoyu Shen asks whether combining these two mechanisms can delay such degradation by distributing the compression burden. They propose a minimalist compound sparsity framework that first applies low-rank approximation and channel pruning to obtain a statically compressed backbone, and then introduces lightweight routers for per-token dynamic layer skipping. This design enables independent control of parameter sparsity and token-level computation sparsity.

Experiments across language understanding and modeling benchmarks show that compound sparsity consistently outperforms single-mechanism compression under the same total sparsity, delaying the decay point on understanding tasks and preserving stronger modeling performance. Further analysis reveals cross-dimensional interference between parameter pruning and token skipping, showing that near-balanced allocation is most effective under a fixed sparsity budget. The results demonstrate that compound compression provides a practical way to improve LLM compression while revealing a broader cross-dimensional sparsity boundary that ultimately limits further compression.