llama.cpp b10083 adds sqrt_softplus CUDA optimization for MoE
New release boosts MoE performance with a custom sqrt_softplus CUDA kernel.
The llama.cpp project, known for its efficient local LLM inference, has tagged release b10083. This version introduces a CUDA kernel for sqrt_softplus, specifically targeting the topk-moe (top-k mixture-of-experts) layer used in models like DeepSeek-V3. The optimization reduces computation overhead by fusing the sqrt and softplus operations on the GPU, which can improve token generation speed and reduce VRAM usage for MoE models running on CUDA-compatible hardware.
The release is accompanied by a broad set of pre-built binaries. Notable builds include macOS Apple Silicon (both standard and KleidiAI-enabled), Intel macOS, Windows x64 with CUDA 12 or CUDA 13 DLLs, Linux with ROCm 7.2, Vulkan, OpenVINO, and SYCL FP16/FP32. Also available are arm64 builds for Ubuntu, Android, and Windows, plus OpenCL for Adreno GPUs. The extensive platform support ensures developers can quickly test the new kernel without compiling from source. The commit is signed with GitHub’s verified signature, maintaining supply chain security.
- CUDA optimization adds sqrt_softplus kernel for topk-moe layers used in dsv4 models
- Release includes builds for over 20 platform/backend combinations including ROCm, Vulkan, and SYCL
- Commit signed with GPG key B5690EEEBB952194 for verified integrity
Why It Matters
Improves local MoE model inference on NVIDIA GPUs, enabling faster and more efficient deployment of advanced LLMs.