HeteroMosaic speeds edge LLMs 2x using CPU, iGPU, and NPU together
New scheduler cuts energy 45% and outperforms single-accelerator baselines by up to 1.78x.
Modern edge SoCs pack CPUs, integrated GPUs (iGPUs), and neural processing units (NPUs) into a single chip, but existing LLM runtimes typically pick one accelerator or optimize operators in isolation. This leaves performance on the table, especially on unified-memory platforms where memory sharing and task-graph coordination matter. A new paper from researchers at the University of Illinois Urbana-Champaign (led by Gregory Hyegang Jun) introduces HeteroMosaic, a heterogeneity-first scheduler that dynamically distributes LLM inference workloads across all available compute units.
HeteroMosaic uses a heterogeneous roofline model to detect when combining iGPU and NPU execution yields benefits, then breaks inference into dependency-preserving micro-batches that can overlap execution across accelerators. It also co-optimizes scheduling and device allocation under real-world constraints like memory contention, DVFS, device variation, and NPU runtime overhead. Implemented in PyTorch C++, tested on three AMD Ryzen AI platforms, HeteroMosaic delivers up to 1.73x speedup over iGPU-only, 1.78x over NPU-only, and 2.05x over frameworks like llama.cpp. Energy savings reach 45.3%, and it beats prior heterogeneous edge AI solutions by up to 2.35x.
- Combines iGPU and NPU execution via micro-batch decomposition for cross-accelerator overlap
- Up to 2.05x faster than existing edge LLM frameworks (llama.cpp) on balanced AMD Ryzen AI platforms
- Reduces energy consumption by up to 45.3% while achieving speedups over single-accelerator baselines
Why It Matters
Enables faster, more efficient LLM inference on edge devices, unlocking new on-device AI applications with lower power draw.