Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLM

A research team of 16, led by Sravanth Kodavanti and primarily affiliated with Samsung, has developed a novel hardware-aware framework that makes deploying versatile large language models (LLMs) on smartphones commercially viable. Their system, detailed in a paper accepted at ACL 2026, is engineered specifically for Samsung Galaxy S24 and S25 devices powered by Qualcomm's SM8650 and SM8750 chipsets. The core innovation is a 'one-for-all' foundational model based on LLaMA that remains frozen, while application-specific Low-Rank Adaptations (LoRAs) are injected as runtime inputs. This allows the model to dynamically switch between 8 distinct tasks—like translation, summarization, or coding—without the memory overhead or need for recompilation typically required when loading separate models.

To drastically improve performance, the team introduced two key acceleration techniques. First, a 'multi-stream decoding' mechanism enables the model to generate multiple stylistic variations of a response—such as formal, polite, or jovial tones—concurrently within a single forward pass, slashing latency by up to 6x for this operation. Second, they applied Dynamic Self-Speculative Decoding (DS2D), a tree-based strategy that allows the model to predict several future tokens without relying on a separate, smaller draft model, achieving a 2.3x speedup in decode time. When combined with aggressive INT4 quantization and other architecture optimizations, the entire framework delivers 4-6x overall improvements in memory usage and latency while maintaining accuracy across 9 languages.