LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning

Researchers Marco Paul E. Apolinario and Kaushik Roy have introduced LANCE (Low-rank Activation Compression), a breakthrough framework designed to make on-device continual learning practical for resource-constrained environments. The core innovation is a one-shot higher-order Singular Value Decomposition (SVD) that creates a reusable low-rank subspace for projecting activations during backpropagation. This eliminates the computational overhead of repeated decompositions used in previous methods, simultaneously reducing both memory and processing requirements. By establishing fixed low-rank subspaces, LANCE also enables continual learning by allocating new tasks to orthogonal subspaces, preventing catastrophic forgetting without storing large, task-specific matrices.

Experimental results demonstrate LANCE's significant efficiency gains. The framework reduces activation storage by up to 250 times while maintaining accuracy comparable to standard full backpropagation across multiple datasets including CIFAR-10/100, Oxford-IIIT Pets, and Flowers102. On continual learning benchmarks like Split CIFAR-100 and Split MiniImageNet, LANCE performs competitively with established orthogonal gradient projection methods, but at a fraction of the memory cost. This positions LANCE as a scalable solution for deploying adaptive AI models directly on smartphones, IoT devices, and other edge hardware, enabling true personalization and long-term adaptation without compromising privacy or requiring constant cloud connectivity.