llama.cpp b10066 adds OpenCL MoE kernel for faster local LLMs
New release optimizes Mixture-of-Experts models with OpenCL on GPUs
Deep Dive
llama.cpp release b10066 incorporates OpenCL loading and usage of the kernel_gemm_moe_q6_k_f32_ns kernel from the bin kernel library (commit #25797). Builds are available for macOS, Linux, Windows, Android, and openEuler across various architectures and backends.
Key Points
- New OpenCL kernel for Mixture-of-Experts (MoE) models in llama.cpp b10066
- Supports multiple platforms: Windows, Linux, macOS, Android, with Vulkan, CUDA, OpenCL, SYCL
- Enables efficient 6-bit quantized MoE inference on non-NVIDIA GPUs like AMD and Intel
Why It Matters
Expands local LLM accessibility to broader GPU hardware via OpenCL, reducing vendor lock-in.