SoLA: Leveraging Soft Activation Sparsity and Low-Rank Decomposition for Large Language Model Compression

Researchers from an academic team have introduced SoLA (Soft activation sparsity and Low-rAnk decomposition), a novel training-free compression technique for large language models. The method addresses the critical deployment challenge of billion-parameter models by analyzing activation patterns in feed-forward networks to identify which components contribute most to inference. SoLA then applies a two-pronged approach: preserving these critical "minority" components while compressing the majority through low-rank decomposition, all without requiring the special hardware support or costly post-training typical of existing methods.

SoLA's key innovation is an adaptive component-wise low-rank allocation strategy that assigns appropriate truncation positions for different weight matrices, significantly reducing decomposition loss. Extensive testing on LLaMA-2 models (7B, 13B, 70B) and Mistral-7B across multiple benchmarks shows remarkable results. At a 30% compression rate on LLaMA-2-70B, SoLA outperforms state-of-the-art methods by reducing perplexity from 6.95 to 4.44—a substantial improvement in language modeling quality—while simultaneously boosting accuracy on downstream tasks by 10%. This represents a breakthrough in making powerful LLMs more accessible for practical deployment.

The technique's training-free nature makes it particularly valuable for organizations without extensive computational resources for fine-tuning. By maintaining or even improving model performance while reducing parameter counts, SoLA opens doors for deploying advanced LLMs on more affordable hardware, in edge computing scenarios, and in cost-sensitive applications. The paper demonstrates that intelligent architectural analysis can achieve compression gains previously thought to require extensive retraining or specialized hardware acceleration.