Researchers prove combining low-rank and quantization degrades LLM performance, propose DAM fix
New mathematical proof shows two popular compression methods aren't orthogonal—causing extra error when combined.
A new paper from researchers at the Chinese Academy of Sciences (Xiusheng Huang, Lu Wang, Yequan Wang, Jun Zhao, Kang Liu) tackles a critical assumption in LLM compression: that low-rank decomposition and quantization can be safely combined without compounding errors. The authors provide the first mathematical proof showing these methods are non-orthogonal—their interaction introduces additional performance degradation beyond each technique's independent cost. This explains why simply stacking compression methods often fails at higher compression ratios.
To solve this, the team introduces the Diagonal Adhesive Method (DAM), a novel approach that allows low-rank and quantization techniques to be merged effectively while minimizing the extra error. Experiments on large language models validate the theory and demonstrate DAM's ability to preserve accuracy even at aggressive compression levels. The paper lays both a theoretical and experimental foundation for future model compression research, offering a clear path to reducing memory and compute requirements without sacrificing model quality—a key need as LLMs continue to scale.
- First mathematical proof that low-rank decomposition and quantization are non-orthogonal in LLM compression
- Combining these methods causes significant performance degradation beyond each method's individual error
- Diagonal Adhesive Method (DAM) enables effective integration of both techniques while mitigating accuracy loss
Why It Matters
Enables more efficient LLM deployment by safely combining compression methods, reducing costs without performance hits.