llama.cpp b10165 adds Vulkan support for iq4_nl and q1_0 quantizations
New release brings faster inference for low-bit quantizations on Vulkan GPUs.
Llama.cpp, the widely-used open-source inference engine for large language models, has released version b10165 with significant Vulkan backend enhancements. The update re-adds iq4_nl quantization support to the flash attention kernel, addressing earlier concerns about shared memory waste that turned out negligible. Additionally, it enables q1_0 support for non-coopmat2 platforms, fixing a previously reported issue (#23681). These improvements target better performance and compatibility for running quantized LLMs on Vulkan-capable GPUs across Linux, Windows, and Android.
The release also continues llama.cpp's tradition of broad platform support, offering binaries for macOS (Apple Silicon and Intel), Linux (x64, arm64, s390x with various backends), Windows (x64, arm64, with CUDA, Vulkan, OpenVINO, SYCL, HIP), and Android arm64. The focus on Vulkan-specific improvements means users with AMD, Intel, or other Vulkan-compatible GPUs can now leverage these low-bit quantizations more efficiently, potentially speeding up local LLM inference and enabling deployment on a wider range of consumer hardware.
- Readds iq4_nl quantization support to Vulkan flash attention kernel, improving inference speed for 4-bit quantized models.
- Adds q1_0 support for non-coopmat2 Vulkan platforms, fixing issue #23681 for broader hardware compatibility.
- Available for Linux, Windows, macOS, Android with multiple backend options including CUDA, ROCm, OpenVINO, and SYCL.
Why It Matters
Enables efficient local LLM inference on Vulkan GPUs, democratizing AI on more consumer hardware.