llama.cpp b9876 fixes crash on MoE models with tensor parallelism + CPU offloading
A tensor parallelism crash in llama.cpp on MoE models is now fixed after a contiguity assertion bug.
The popular open-source C++ inference engine llama.cpp has released version b9876, addressing a critical crash that affected users running mixture-of-experts (MoE) models with tensor parallelism. Specifically, when combining the -sm tensor flag (split mode tensor) with -ncmoe (CPU-offloaded MoE experts), the GGML backend would abort during warm-up due to a GGML_ASSERT(ggml_is_contiguous(tensor)) failure in ggml-backend-meta.cpp.
The root cause involved the MoE router output tensor (ffn_moe_topk), which is mirrored across backends (GGML_BACKEND_SPLIT_AXIS_MIRRORED) to ensure identical routing decisions. This tensor happened to be a non-contiguous view. The ggml_backend_meta_buffer_{get,set}_tensor functions asserted contiguity before checking the split state, so mirrored non-contiguous tensors incorrectly triggered the assertion. The fix, credited to reporter nathanmp, simply moves the split-state lookup above the contiguity check, allowing the mirrored case to pass through. This release is available across all major platforms including macOS Apple Silicon, Linux (CPU, Vulkan, ROCm, etc.), Windows (CPU, CUDA, Vulkan), and Android.
- Fixes GGML_ASSERT crash in llama.cpp when using tensor parallelism with CPU-offloaded MoE experts (-ncmoe).
- Issue #24886 caused by non-contiguous mirrored router tensors tripping a contiguity assertion before split-state handling.
- Fix moves split-state lookup above the assert in ggml_backend_meta_buffer_{get,set}_tensor functions.
Why It Matters
Makes llama.cpp stable for large MoE models under distributed inference, a common setup for cutting-edge AI workloads.