llama.cpp b9859: OpenCL now loads precompiled binary kernels
New release enables faster GPU inference on OpenCL devices with precompiled kernel libraries
The latest llama.cpp release (b9859) focuses on OpenCL acceleration, enabling the loading of precompiled binary kernels from external libraries. This improvement addresses a cyclic dependency between ggml-opencl and ggml-backend-dl by introducing libdl.h, a lightweight dynamic linking helper. Developers can now package precompiled kernel binaries alongside their applications, allowing llama.cpp to load optimized kernels at runtime without recompilation. The update covers multiple kernel types: gemm_moe_mxfp4_f32_ns (for mixture-of-experts mixed-precision), q8_0 and q4_0/q4_1 general matrix multiply (GEMM), and quantized kernels (q4_k) for MoE operations. These kernels are loaded via function pointer callbacks, with fallback to JIT compilation if the binary library is unavailable. The change also includes updated documentation and improved error messages, and supports platforms like Windows x64 (OpenCL Adreno), macOS, Linux (Vulkan/ROCm), and Android arm64. Users running AI models on AMD GPUs, Adreno mobile GPUs, or other OpenCL-compatible hardware will see faster inference times and lower initialization latency.
- Introduces precompiled binary kernel loading for OpenCL, reducing runtime compilation overhead
- Adds libdl.h to break cyclic dependency between ggml-opencl and ggml-backend-dl
- Supports specific kernels: gemm_moe_mxfp4_f32_ns, q8_0 gemm, q4_0/q4_1/q4_k moe gemm from binary libraries
Why It Matters
Faster AI inference on OpenCL devices (AMD, Adreno) with precompiled kernels, reducing startup time and improving performance.