llama.cpp gains 3.6x speedup for Q2_0 models on x86 CPUs
New VNNI-optimized Q2_0 kernel slashes 8B model decode latency by 3.4x
The community-driven llama.cpp project merged a pull request (#26348) that introduces a hardware-accelerated path for the Q2_0 × Q8_0 dot product using AVX-VNNI or AVX-512 VNNI instructions on x86 CPUs. Unlike incremental kernel tweaks, this change replaces the generic implementation with a vector neural network instruction (VNNI) optimized routine, resulting in a 3.0–3.6x throughput uplift across models ranging from 1.7B to 27B parameters.
Tested on an AMD EPYC 9645 (8 cores, AVX-512 enabled), an 8B Q2_0 model’s decode throughput rose from 2.39 tok/s to 8.20 tok/s (+3.43x). On an Intel i5-13400—where AVX-512 is fused off despite AVX-VNNI being present—the same model jumped from 2.17 tok/s to 6.92 tok/s (+3.19x). The optimizations are model- and quantization-specific; Q4/Q5 quantizations remain unaffected. While promising, the gains are not universal: the PR is still open, upstream benchmarks focused on server-grade silicon, and real-world performance on mobile hardware remains untested. The author verified numerical correctness with 14,000 randomized kernel comparisons and showed identical top-token selection in 99.2% of inference runs.
- llama.cpp PR #26348 adds AVX-VNNI/AVX-512 VNNI acceleration for Q2_0 × Q8_0 dot products, delivering 3–3.6x speedups on x86 CPUs
- 8B Q2_0 decode jumps from 2.39 tok/s → 8.20 tok/s on AMD EPYC 9645; Intel i5-13400 sees 2.17 tok/s → 6.92 tok/s
- Speedups are quantization-specific (Q2_0 only), the PR is still open, and mobile CPU performance remains unverified
Why It Matters
This optimization could slash inference latency for quantized LLMs on mainstream laptops and workstations, making edge AI more practical without hardware upgrades.