Google’s Gemma 4 QAT reaches 71 tok/s decode on Strix Halo APU
QAT preserves model accuracy while running Gemma 4 locally on AMD’s latest APU.
Deep Dive
Google's Gemma 4 QAT (quantization-aware training) models were benchmarked on an AMD Ryzen AI Max+ 395 Strix Halo APU using llama.cpp Vulkan. The 26B-A4B Q4_0 GGUF achieved 1,194.4 tok/s prefill with plain F16 KV and 71.4 tok/s decode with QAT MTP + Q8 KV. With the QAT MTP + Q8 KV setup, prefill was 729.3 tok/s and decode 71.4 tok/s. Models include 12B, 26B-A4B, and 31B, all running on 128GB unified LPDDR5X. QAT aims to preserve more of the original model's behavior than standard post-training quantization.
Key Points
- Gemma 4 26B-A4B Q4_0 achieved 71.4 tok/s decode with QAT MTP + Q8 KV on AMD Strix Halo APU.
- QAT (quantization-aware training) preserves more original model behavior than standard post-training quantization.
- Three model sizes tested: 12B, 26B-A4B, and 31B, all fitting within 16.44 GiB on 128GB unified memory.
Why It Matters
High-quality local LLM inference on consumer APUs reduces cloud dependency and enables private, low-latency AI.