llama.cpp b10067 fixes DeepSeek-V4 quantization crash
DeepSeek-V4 models now quantizable on CPU/GPU after i32 tensor fix
llama.cpp, the popular open-source C/C++ library for local LLM inference, released b10067 on July 18. This patch resolves a quantization error exclusively affecting DeepSeek-V4's Mixture-of-Experts architecture. The tensor ffn_gate_tid2eid is an integer index mapping token IDs to expert IDs, not a floating-point weight matrix. During quantization, the software attempted to convert these integers to float, crashing. The fix adds this tensor to an exclusion list, letting quantization proceed normally.
Now users can quantize DeepSeek-V4 models to smaller precision formats (e.g., Q4_K_M) using llama-quant, drastically reducing memory requirements while retaining quality. The release also verifies the commit with GitHub's GPG signature. Pre-built binaries are available across 20+ platforms: macOS (Apple Silicon and Intel), Linux (x64, arm64, s390x with Vulkan, ROCm 7.2, OpenVINO, SYCL), Windows (CPU, ARM, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), Android (arm64), and openEuler (with ACL Graph). This fix unlocks local, efficient deployment of one of the largest open-weight MoE models.
- Fixes crash when quantizing DeepSeek-V4 due to i32 routing table tensor being misclassified as float weights
- Excludes ffn_gate_tid2eid alongside ffn_gate_inp.weight from quantization in llama-quant
- Release supports 20+ platform builds including Apple Silicon, Windows CUDA, Linux ROCm, Android, and openEuler
Why It Matters
Enables local high-performance inference of DeepSeek-V4 MoE models on consumer hardware