Rejected llama.cpp PR boosts Strix Halo MOE performance by 30%

A rejected pull request (PR) by developer pedapudi for the llama.cpp project delivers a significant performance boost for Mixture of Experts (MOE) models on AMD hardware, specifically the Strix Halo iGPU. The PR, denied for mainline inclusion, offers up to 31% faster prompt processing (PP) at low context sizes. Benchmarks on a Strix Halo system with 128GB VRAM show the Qwen 35B A3B Q4_K model achieving 1448 tokens per second (t/s) at 512 context, compared to 1106 t/s without the patch. The improvement stems from optimized kernel handling for MOE architectures, applied via a small code change that users can manually insert into the current llama.cpp release.

The gains are highly context-dependent. At 10,000 tokens, speedup drops to 20%; at 20,000 it's 16%; at 40,000 it's 11%; and at 60,000 it falls to 8%. Pedapudi explains in the PR that the diminishing returns are due to memory bandwidth bottlenecks becoming dominant with larger contexts. The patch only affects models using MOE (mixture of experts) layers, common in efficient large language models like Qwen and Mixtral. For Strix Halo owners running local LLMs, this manual tweak can substantially improve inference throughput, especially for short to medium-length prompts.