llama.cpp b10058 adds Vulkan support for Q2_0 quantization
New release enables faster AI inference on Vulkan GPUs with Q2_0 models.
The llama.cpp project, maintained by ggml-org, has released version b10058, which introduces Vulkan support for the Q2_0 quantization format. Q2_0 is a 2-bit quantization method that drastically reduces model size and memory footprint, making it ideal for running large language models on consumer hardware. The backend performance tests for matrix-vector multiplication initially showed poor results compared to the existing Q2_K quantization, but the team improved performance by doubling the number of rows per workgroup. This adjustment led to a significant speedup, making Vulkan a viable backend for Q2_0 inference.
The release includes pre-built binaries for a wide range of platforms: macOS (Apple Silicon and Intel), Linux (CPU and Vulkan), Windows (CPU, Vulkan, CUDA, OpenCL), Android, and various specialized backends like ROCm and OpenVINO. This broad support ensures that developers and researchers can leverage Vulkan-accelerated Q2_0 inference across nearly any device with a Vulkan-compatible GPU. For users, this means faster, more efficient local LLM inference without relying on proprietary GPU APIs like CUDA.
- Vulkan support for Q2_0 (2-bit) quantization in llama.cpp b10058
- Improved mat-vec-mul performance by doubling rows per workgroup
- Available on multiple platforms including macOS, Linux, Windows, Android, and more
Why It Matters
Expands efficient local LLM inference to Vulkan GPUs, democratizing AI beyond NVIDIA hardware.