Developer Tools

llama.cpp b9858 fixes model path for Hugging Face shards

New release resolves model loading issues for split HF models in llama.cpp

Deep Dive

llama.cpp, the popular open-source C/C++ library for LLM inference by ggml-org, has released version b9858. This minor update addresses a specific bug (issue #25181) where the model path resolution failed for Hugging Face models stored in multiple shards (split files). The fix modifies the common module to use the primary split file (typically the first shard) as the base model path, aligning with how Hugging Face structures multi-part model weights. This ensures that llama.cpp correctly identifies and loads all necessary shards without manual intervention. The release ships with pre-built binaries across a wide range of platforms, including macOS (Apple Silicon and Intel), Linux (x64, ARM64, s390x, with support for Vulkan, ROCm 7.2, OpenVINO, SYCL), Windows (CPU, ARM64, OpenCL Adreno, CUDA 12 and 13, Vulkan, OpenVINO, SYCL, HIP), and Android ARM64. Notably, the KleidiAI-enabled macOS build and several openEuler builds remain disabled in this release.

While a small patch, this update directly improves the user experience for anyone running large language models from Hugging Face that are split across multiple tensors—a common practice for models above consumer GPU memory limits. By automatically resolving the primary shard, llama.cpp reduces friction for developers and researchers deploying models like Llama 3, Mistral, or Gemma locally. The fix also reinforces llama.cpp's commitment to broad compatibility with the Hugging Face ecosystem, which remains the dominant hub for open-weight models. Users on any supported platform can upgrade to b9858 via the GitHub releases page or by pulling the latest commit from the repository.

Key Points
  • Fixes issue #25181: model path now uses Hugging Face primary split automatically
  • Applies to all supported platforms: macOS, Linux, Windows, Android, and more
  • Ensures correct loading of multi-shard models (e.g., Llama, Mistral) without manual path fixes

Why It Matters

Seamless loading of sharded HF models improves local LLM deployment for professionals and enthusiasts.

📬 Get the top 10 AI stories daily