b8352
The popular open-source inference engine now supports Alibaba's latest models with new 4-bit floating point precision.
The open-source community behind llama.cpp, the widely-used C++ inference engine for running large language models locally, has released a significant update with commit b8352. This release primarily focuses on adding comprehensive support for Alibaba's Qwen3.5 and Qwen3.5MoE model families by implementing NVFP4 (4-bit floating point) tensor support through pull request #20506. The update represents a crucial integration that allows these competitive Chinese models to run efficiently alongside established options like Llama and Mistral within the same optimized framework.
Beyond the Qwen integration, the release maintains llama.cpp's hallmark cross-platform compatibility with pre-built binaries for macOS (both Apple Silicon and Intel), Windows (including CUDA 12/13, Vulkan, and SYCL backends), Linux (with CPU, Vulkan, ROCm 7.2, and OpenVINO options), and even specialized builds for openEuler with Huawei Ascend NPU support. The NVFP4 quantization specifically enables users to run larger Qwen models with reduced memory footprint while maintaining reasonable accuracy, making advanced Chinese-language AI more accessible on consumer hardware without requiring enterprise-grade GPUs.
- Adds NVFP4 (4-bit floating point) tensor support for Alibaba's Qwen3.5 and Qwen3.5MoE models via PR #20506
- Maintains cross-platform binaries for Windows CUDA, macOS Apple Silicon, Linux ROCm, and openEuler with Ascend NPU support
- Enables more efficient local inference of state-of-the-art Chinese language models on consumer hardware
Why It Matters
Democratizes access to competitive Chinese LLMs by enabling efficient local inference, expanding the open-source ecosystem beyond Western models.