llama.cpp b9521 boosts MTP decoding with CUDA PDL optimization
Up to 10% faster speculative decoding on AMD B4500 GPUs
llama.cpp, the popular C++ library for running LLMs locally, released version b9521 with a key CUDA optimization. The commit enrolls the sparse MoE (Mixture of Experts) kernel mul_mat_vec_q_moe into PDL (performance-driven loop), which reorganizes memory access patterns for better GPU utilization. Benchmarks on a B4500 GPU (likely an AMD Instinct) show consistent improvements in MTP (multi-token prediction) speculative decoding across nine tasks. For example, summarization speed rose from 226.6 to 240.2 tokens per second (a 6% gain), question answering from 225.1 to 238.5 tok/s, and code C++ from 212.8 to 224.6 tok/s. The average improvement across all tasks is approximately 5-7%.
This release continues llama.cpp's tradition of supporting a wide range of hardware. The b9521 binary is available for macOS (Apple Silicon and Intel), iOS, Linux (x64, arm64, s390x, with Vulkan, ROCm 7.2, OpenVINO, SYCL), Android (arm64), and Windows (x64 and arm64 with CUDA 12/13, Vulkan, HIP). For developers running local LLMs with speculative decoding, this optimization directly reduces latency and increases throughput without requiring model changes.
- Enrolls mul_mat_vec_q_moe kernel into PDL for CUDA GPUs, improving memory access patterns
- Benchmarks on B4500 show 5-10% tok/s gains across MTP tasks (e.g., summarization: 226.6→240.2 tok/s)
- Supports 20+ platform builds including Windows CUDA 13, Linux ROCm, macOS Apple Silicon, and Android
Why It Matters
Faster local LLM inference with speculative decoding means lower latency for real-time AI applications on consumer GPUs.