llama.cpp adds Cohere2-MoE support in b9626 release
Now run Command R+ MoE models locally with C++.
The llama.cpp project, known for running large language models on consumer hardware, shipped version b9626 that introduces first-class support for Cohere's latest mixture-of-experts architecture, dubbed cohere2-MoE. The release refactors how gating and expert selection work, switching from redundant checks to a cleaner sigmoid mapping, and renames the architecture tag from 'cohere2-moe' to 'cohere2moe'. Key fixes include resolving 'first_k_dense_replace' errors in Transformers, handling shared expert averaging with a (routed+shared)*0.5 formula, and correcting sliding window attention patterns. The update also supports North-Mini-Code-1.0, a smaller Cohere variant, and forces metadata usage for better model loading.
Cross-platform compatibility remains a highlight: automated CI builds are provided for macOS (Apple Silicon with optional KleidiAI acceleration, Intel x64, and iOS XCFramework), Linux (x64/arm64 CPU, Vulkan, ROCm 7.2, OpenVINO, Intel SYCL FP32/FP16), Windows (x64/arm64 CPU, CUDA 12/13, Vulkan, SYCL, HIP), Android arm64 CPU, and even openEuler with ACL Graph. All build artifacts are available on the release page. This release cements llama.cpp as the go-to runtime for testing Cohere's latest MoE models locally, giving developers and researchers zero-cost access to state-of-the-art architectures.
- Adds architecture support for Cohere's cohere2-MoE (Command R+) with refined expert gating and transformer fixes.
- Fixes multiple issues: sliding window attention, MTP failures, and Transformers crash on 'first_k_dense_replace'.
- Ready-to-use builds for macOS, Linux, Windows, Android, and iOS across CPU/GPU backends (CUDA, Vulkan, ROCm, SYCL).
Why It Matters
Brings cutting-edge MoE models like Command R+ to local inference, reducing cloud dependency for AI workloads.