llama.cpp b9286 adds Q8_0 quantization for ZenDNN backend
New release speeds up LLM inference on AMD ZenDNN with 8-bit quantization.
llama.cpp, the popular C++ library for running large language models (LLMs) locally, has released version b9286 with a key enhancement: Q8_0 quantization support for the ggml-zendnn backend. Q8_0 is an 8-bit quantization scheme that compresses model weights from 16-bit to 8-bit, reducing memory usage by nearly half while maintaining most of the model's accuracy. By implementing this for the ZenDNN (AMD’s deep neural network library) backend, users with AMD CPUs or accelerators can now run LLMs more efficiently—faster token generation and lower memory footprint.
The release also includes various platform-specific binaries: macOS (Apple Silicon and Intel, with optional KleidiAI), Linux (x64, arm64, s390x with Vulkan, ROCm, OpenVINO, SYCL support), Windows (x64 and arm64 with CUDA, Vulkan, SYCL, HIP), and Android arm64. Additionally, openEuler builds for x86 and aarch64 with ACL Graph support are available. This broad compatibility ensures that developers can deploy Q8_0-optimized models across diverse hardware, from personal laptops to servers.
- b9286 adds Q8_0 8-bit quantization specifically for the ggml-zendnn backend
- Reduces memory usage ~50% and accelerates inference on AMD ZenDNN-supported hardware
- Includes prebuilt binaries for over 20 platform/accelerator combinations (CUDA, Vulkan, ROCm, etc.)
Why It Matters
Q8_0 quantization on ZenDNN makes local LLM inference faster and more accessible for AMD hardware users.