Llama.cpp b9735 Just Unlocked a Hidden 47% Speed Boost on Your CPU — Here's How
New LLM runtime optimization speeds prompt processing by up to 1.47x on Intel Xeon
Llama.cpp, the widely-used open-source library for running LLMs locally on consumer hardware, just shipped version b9735 with a critical performance optimization. The release, tagged by maintainers on GitHub, introduces AMX (Advanced Matrix Extensions) support — a set of Intel-specific instructions designed to accelerate matrix operations common in neural network inference. The change flattens the partition over n_batch * M so that every thread participates in quantization, improving throughput on CPUs with AMX hardware.
Benchmarks run on an Intel Xeon Platinum 8488C show dramatic improvements: prompt processing (pp512) for the Q4_K_S quantization variant jumped from 732.29 tokens/s to 1077.29 tokens/s — a 47% boost. Other quantizations like Q4_K_M and IQ4_XS saw gains of 40% and 41% respectively. Token generation speeds remained stable (within 2% of previous performance), meaning real-world interactive use cases like chatbots also benefit from faster context ingestion. The patch was contributed by Adrien Gallouët from Hugging Face and is now available for all platforms in the latest build.
- AMX (Advanced Matrix Extensions) optimization speeds up batch prompt processing on Intel Xeon Platinum CPUs by up to 47%
- Largest gains seen in Q4_K_S quantization: 1077 t/s vs 732 t/s (old) for pp512 on Xeon 8488C
- Patch authored by Hugging Face engineer Adrien Gallouët; included in llama.cpp release b9735
Why It Matters
Faster local LLM inference on high-end Intel CPUs reduces latency for AI applications running on-premises or in edge deployments.