b8579

The open-source llama.cpp project, maintained by ggml-org, has released a significant performance update with commit b8579. This commit specifically targets the computational kernel for Mixture of Experts (MoE) models, such as Mistral AI's Mixtral. The core improvement is a new 'mul_mat_vec_q_moe' kernel designed for multi-token processing (batch size > 1). The previous kernel suffered from inefficiency, spawning too many thread blocks (nrows_x, nchannels_dst, ncols_dst) where each block did minimal work—a block size of (32, 4) handled just a single row's inner dot product.

The new architecture restructures the workload onto a grid of (ceil(nrows_x/rpb), nchannels_dst) with a block size of (warp_size, ncols_dst). Crucially, each warp now processes two rows independently and performs reductions at the warp level, completely avoiding the latency cost of synchronizing data through shared memory. This change simplifies the codebase by removing the 'is_multi_token_id' specialization and does not increase compilation time. The update also cherry-picks optimizations from contributor @am17an, makes the max batch size for these kernels configurable based on GPU architecture, and increases the max batch size for MMVQ kernels to 8.

This low-level optimization is a prime example of the relentless performance tuning that makes llama.cpp a cornerstone for efficient local AI inference. By squeezing more operations into warps and avoiding synchronization barriers, the update directly translates to faster throughput for users running advanced MoE models on their own hardware, from Apple Silicon Macs to CUDA-enabled NVIDIA GPUs.