New NPU framework speeds on-device diffusion LLMs by 17–42x
Researchers accelerate LLaDA-8B up to 42x on smartphones with mobile NPU optimizations.
Diffusion large language models (dLLMs) generate text by denoising multiple tokens in parallel, promising low-latency inference on smartphones. However, the repeated denoising steps create massive computation that overwhelms mobile CPUs. Mobile neural processing units (NPUs) can handle dense matrix operations efficiently, but they struggle with dLLM workloads due to shrinking per-block workloads, complicated KV cache reuse, and limited address space causing costly data transfers.
A team of researchers (Wang, Sun, Ren) introduces the first NPU-aware inference framework for dLLMs on smartphones, addressing these challenges with three innovations. Multi-Block Speculative Decoding fills shrinking compute blocks with future tokens to maximize NPU utilization. Dual-Path Progressive Revision allows committed tokens to be revised until stable using a CPU side path without stalling NPU execution. Swap-Optimized Memory Runtime compacts address layouts and overlaps data staging with NPU computation to reduce overhead. In tests on LLaDA-8B across diverse hardware, the framework reduces generation latency by 17x to 42x compared to a CPU baseline with prefix KV cache reuse, while maintaining output quality.
- Achieves 17x–42x latency reduction on LLaDA-8B vs. CPU baseline with prefix KV cache reuse.
- Introduces three techniques: Multi-Block Speculative Decoding, Dual-Path Progressive Revision, and Swap-Optimized Memory Runtime.
- First NPU-aware framework specifically designed for diffusion LLM inference on mobile devices.
Why It Matters
Brings real-time diffusion LLM performance to smartphones, enabling responsive on-device AI without cloud latency.