AMC cuts transformer energy 59% with saliency-driven compression
Save 59% energy on edge transformers with only 3.6% accuracy loss.
Deploying large transformer models on resource-constrained edge devices is notoriously difficult due to the uniform processing of all tokens, which wastes energy on simpler inputs. A new paper from researchers (Jiayin Hu, Kai Yuan, et al.) proposes Adaptive Model Compression (AMC), a framework that uses saliency—a measure of token importance—to guide dynamic resource allocation. In a multi-tier architecture, high-saliency tokens are processed at full precision, while less critical tokens use reduced rank and lower bit-width representations. This targeted approach cuts computational waste without sacrificing overall output quality.
Experimental results on 45nm CMOS hardware demonstrate that AMC reduces system energy by 59.2% and boosts throughput by 2.24x. The trade-off is a marginal 3.6% drop in accuracy. Such efficiency gains could significantly extend battery life in mobile and IoT devices, making on-device LLM inference practical. The work bridges the gap between state-of-the-art AI and edge deployment, offering a path toward smarter, longer-lasting embedded systems.
- AMC uses saliency-driven resource allocation to assign full precision only to important tokens.
- Achieves 59.2% energy reduction and 2.24x throughput increase on 45nm CMOS hardware.
- Maintains robust performance with only a 3.6% accuracy trade-off compared to static inference.
Why It Matters
Makes large transformer models feasible on battery-powered devices, enabling smarter edge AI with longer operational life.