Unsloth's NVFP4 quants beat NVIDIA's Qwen3.6 by up to 2.5x
Local LLM inference gets 2.5x faster on Qwen3.6-27B with no accuracy loss.
Unsloth has open‑sourced custom NVFP4 quantizations for the newly released Qwen3.6 family, claiming significant speedups over NVIDIA’s own NVFP4 implementations. For the 27B model, their quant runs 2.5× faster; for the 35B-A3B mixture‑of‑experts model, they offer two variants: “NVFP4” (1.56× faster) and “NVFP4‑Fast” (1.79× faster). The key innovation is using W4A4 (4‑bit weights and 4‑bit activations) for matrix multiplies, while NVIDIA’s quants use W4A16 (4‑bit weights, 16‑bit activations). This fully leverages actual 4‑bit tensor core hardware, boosting throughput without sacrificing quality.
Accuracy benchmarks confirm parity. On MMLU‑Pro, GPQA, and AIME 2025, Unsloth’s quants match or slightly exceed both NVIDIA’s NVFP4 and FP8 / BF16 baselines. Additionally, the release includes FP8 KV cache calibration, which automatically enables 2× longer context windows – a major win for long‑document or multi‑turn applications. Unsloth also pre‑embeds Multi‑Token Prediction (MTP) support. For users on DGX Spark hardware, they warn to use the FlashInfer backend or risk 2× slower inference. Full benchmarks and setup guides are available in their blog post.
- Unsloth's NVFP4 quants for Qwen3.6-27B achieve 2.5x faster inference than NVIDIA's version using W4A4 tensor cores.
- Two variants for 35B-A3B: 'NVFP4' (1.56x faster) and 'NVFP4-Fast' (1.79x faster) with negligible accuracy trade‑off.
- FP8 KV cache calibration extends context length by 2x automatically; accuracy matches or exceeds FP8/BF16 baselines on MMLU-Pro, GPQA, AIME 2025.
Why It Matters
Faster, lossless local LLM inference on consumer hardware – a game changer for developers running Qwen3.6 offline.