New LLM Compression Method Combines Neuron Importance and Data-Aware Low-Rank Approximation

Large language models (LLMs) boast billions of parameters but suffer from massive memory footprints, limiting their use on edge devices or in low-resource settings. Researchers from the National Technical University of Athens now propose a compression method that fuses two previously isolated approaches: neuron importance (identifying which weights are critical for task performance) and data-aware low-rank approximation (using SVD to decompose weight matrices with input data to minimize error). By combining these into a single objective, they achieve higher fidelity at extreme compression rates. Additionally, they introduce a computationally efficient algorithm for dynamic compression rate allocation across layers, replacing the usual uniform or heuristic-based approaches. This method automatically figures out which layers can tolerate more compression and which need more precision, all without exhaustive search.

Experimental results, published in IEEE Access, show that the approach performs on par or substantially better than previous state-of-the-art, especially under high compression ratios (e.g., reducing model size by 90-95%). The gains are most pronounced when models are compressed aggressively, a regime where older methods often cause sharp accuracy drops. The technique works on standard transformer architectures and can be applied to popular open-source LLMs. This means developers can now run a 70B-parameter model on a single consumer GPU, or deploy a 7B model on a phone, with only a few percentage points of accuracy loss. The code is expected to be released, offering a practical tool for efficient LLM deployment.