b8944

llama.cpp's latest release (b8944) introduces 64-byte aligned tile buffers, a low-level memory optimization that boosts inference performance on Qwen 3.5B models by up to 3% in prompt processing (pp512) and token generation (tg128). The update, signed by Hugging Face's Adrien Gallouët, shows consistent gains across quantizations: IQ4_NL at 4.5 bpw saw a 3% speedup (82.39 to 84.46 tg128), Q4_K_M improved 1% (76.59 to 77.06 tg128), and Q8_0 saw a 2% uplift in prompt processing. The optimization targets memory alignment to reduce cache misses, particularly benefiting Apple Silicon (arm64 with KleidiAI), Linux (x64, Vulkan, ROCm, OpenVINO), Windows (CUDA 12/13, Vulkan), and Android arm64. No regressions were observed in the 24 tested configurations.

This release continues llama.cpp's trajectory as the leading open-source inference engine for local LLMs, now with 107k stars and 17.4k forks. The performance improvements, while modest, compound across long-running workloads like chatbots, code assistants, and document processing. The update also includes pre-built binaries for macOS, Linux, Windows, Android, and openEuler, making it easy to deploy on diverse hardware. For developers running Qwen, Llama, or Mistral models locally, this release offers a free, low-effort speed boost without model changes.