b8164

The open-source llama.cpp project, maintained by ggml-org, has released a significant technical update with commit b8164. This commit introduces a new optimization for running Mixture of Experts (MoE) models—a popular architecture used in models like Mixtral and DeepSeek—by adding an option to merge the gate and expert weights during the conversion from Hugging Face format to GGUF. This change, contributed by developer Sigbjørn Skjæret, represents a meaningful performance improvement for the widely-used inference engine that enables running large language models on consumer hardware.

The technical implementation modifies the convert_hf_to_gguf.py script to consolidate weight matrices, specifically targeting the 'gate_up' tensors in MoE models. By merging these components, the update reduces memory overhead and computational complexity during inference, potentially offering 20-30% faster processing for MoE-based models. The change is available across all supported platforms including Apple Silicon, CUDA, Vulkan, and ROCm backends, making it immediately accessible to the project's 96k GitHub stars. This optimization demonstrates the continued refinement of efficient inference techniques as MoE architectures become increasingly prevalent in both open and proprietary models.