b8169
The commit fixes AMX support and adds batching, cutting prompt eval time by 18% on Apple chips.
The open-source llama.cpp project, maintained by ggml-org, has released a significant performance update with commit b8169. This technical patch specifically addresses AMX (Advanced Matrix Extensions) support on Apple Silicon processors (M-series chips) and introduces batched processing capabilities. The commit, signed by Adrien Gallouët of Hugging Face, resolves a critical optimization path for Intel's AMX instructions that are emulated on Apple's ARM architecture, allowing for more efficient matrix operations crucial for transformer inference. The release notes include comprehensive before-and-after benchmarks run on the Qwen3-0.6B-GGUF model, demonstrating tangible speedups.
The technical improvements are substantial: prompt evaluation time decreased from 2037.82ms to 1676.23ms (an 18% improvement) for processing 4096 tokens, boosting throughput from 2009.99 to 2443.58 tokens per second. Total processing time for the benchmark dropped by 33%, from 6403ms to 4258ms. Crucially, the update eliminates the separate 'CPU_REPACK' memory allocation (288 MiB), consolidating it into the AMX memory segment, simplifying memory management. The identical perplexity score of ~21.82 confirms the performance gains don't sacrifice output quality. This optimization is part of llama.cpp's ongoing mission to deliver efficient, cross-platform LLM inference, with pre-built binaries available for macOS, Linux, Windows, and openEuler.
- Prompt evaluation time improved by 18% (2038ms to 1676ms) for 4096 tokens on Apple Silicon
- Total processing time reduced by 33% (6403ms to 4258ms) in Qwen3-0.6B benchmarks
- Memory management simplified by eliminating CPU_REPACK segment, consolidating 288 MiB into AMX memory
Why It Matters
Faster local LLM inference on Apple hardware enables more responsive AI applications and efficient model deployment for developers.