TileFuse brings AWQ quantized LLMs to AMD XDNA2 NPUs with 2x speedup

A team of researchers from the University of Illinois at Urbana-Champaign has introduced TileFuse, a close-to-metal mixed-precision kernel library designed specifically for AMD XDNA2 NPUs. The library tackles a key challenge in on-device LLM inference: popular quantization formats like AWQ (W4A16 and W8A16) don't map cleanly onto existing NPU software stacks, which are often proprietary and restrict low-level control. TileFuse co-designs weight layout, metadata placement, and dataflow to bring these off-the-shelf quantization schemes directly to XDNA2 hardware, rather than forcing models to adopt NPU-specific quantization.

TileFuse fuses unpacking, dequantization, and GEMM/GEMV execution into a single kernel flow, introducing an interleaved pre-tiling layout that supports GEMM dimensions up to 32K and redesigning GEMV dataflow to utilize the full 4x8 AIE array. In kernel-level evaluations, TileFuse delivers up to 121.6% performance improvement for GEMM and 281% for GEMV over full-precision baselines, while achieving more than 2x performance and energy-efficiency gains over strong iGPU baselines. End-to-end tests on Ryzen AI laptops show up to 2.0x lower prefilling latency and over 64.6% lower energy consumption, demonstrating that XDNA2 NPUs can be a practical target for AWQ-style edge LLM inference.