llama.cpp b9866 boosts MoE decode by 2.4% with 288-expert support

llama.cpp, the popular C/C++ inference engine for local LLMs, has shipped release b9866 with a targeted optimization for mixture-of-experts (MoE) models. The key change enables CUDA topk-MoE fusion for exactly 288 experts — a count that previously fell back to an unfused per-layer routing chain involving softmax/sigmoid, argsort, get_rows, sum_rows, div, clamp, and scale. At batch size 1, that unfused path added roughly 330 extra graph nodes per token. The new release adds the missing template instantiation and eligibility check, allowing the existing fusion kernel (which already handles multiples of the warp size) to work with 288 experts.

The performance impact is measurable but context-dependent. Benchmarked on a gfx1151 GPU with the Step-3.7-Flash model at IQ4_XS quantization, prompt processing (pp4096) remained unchanged at ~461 tokens/s. Decode performance (tg128) improved from 19.10 to 19.56 tokens per second — a +2.4% gain. However, at deeper contexts (30k tokens), the gain vanishes because decode becomes attention-bound over the KV cache, making the fixed routing overhead negligible. The optimization primarily benefits early-context decode throughput, which is critical for interactive chat and real-time generation scenarios.