Qwen3.5 122B MoE gets 36% faster inference on AMD with ROCmFP4
New AMD‑friendly quantization runs a 122B model at 28.5 tok/s using only 60.7 GiB.
Deep Dive
RedParaglider posted a quantized model using charlie12345/ROCmFPX, won't work on native llama.cpp yet. 122B total · 10B active · 60.70 GiB · 28.50 tok/s MTP-off · BF16 KLD 0.041366 · Decode 28.505 Decode speed + 36.89% faster Size - 13.47gb smaller.
Key Points
- Qwen3.5 122B MoE: 122B total parameters, only 10B active per token, reducing compute load.
- ROCmFP4 quantization achieves 28.50 tok/s decode speed on AMD GPUs with 60.70 GiB memory.
- 36.89% faster and 13.47 GB smaller than previous quantization methods, but requires custom ROCm build.
Why It Matters
AMD GPU users gain a practical 122B‑class MoE model at 28 tok/s, narrowing the gap with NVIDIA for local inference.