Developer Tools

llama.cpp b9931 speeds MoE models with OpenCL tile-skip optimization

New release skips padded tiles in MoE prefill, offering up to 2x faster GEMM.

Deep Dive

The llama.cpp team has released b9931, which brings a significant OpenCL optimization for Mixture-of-Experts (MoE) models. The key change is a 'ragged-tile MoE prefill FP16 GEMM' optimization that skips padded expert tiles. MoE prefill groups tokens into tiles of 32; when the upper 16 slots are all padding (router index 0xFFFFFFFF), the second half of the dot product is skipped. This cuts compute almost in half for low token-per-expert loads, which is common during inference.

Further refinement adds quarter-granularity skip: instead of skipping only half-tiles, the kernel now divides tiles into 8-column groups and skips each empty trailing group independently. This is controlled via the GGML_OPENCL_MOE_RAGGED_GRAN environment variable (options: 8 for quarter, 16 for half, 32 for off; default is 8). The optimization is applied to all eight *_f32_ns MoE GEMM variants and produces byte-identical results, ensuring no accuracy loss. The release also packages builds for multiple platforms including macOS (Apple Silicon, Intel), Linux (x64, arm64, s390x, Vulkan, ROCm, OpenVINO, SYCL), Windows (x64, arm64, CUDA, Vulkan, OpenVINO, HIP), Android (arm64 CPU), and iOS (XCFramework).

Key Points
  • OpenCL MoE prefill GEMM skips padded 16-slot tile halves, reducing compute for low token-per-expert.
  • Quarter-granularity skip (8-col groups) via GGML_OPENCL_MOE_RAGGED_GRAN env var, default quarter.
  • Optimization applied to eight *_f32_ns MoE GEMMs; byte-identical results, no accuracy loss.

Why It Matters

Faster MoE inference on OpenCL devices means cheaper LLM serving and broader hardware support.

📬 Get the top 10 AI stories daily