b8878
The open-source project now enables efficient AI on Snapdragon chips, expanding beyond Apple and Nvidia.
The open-source community behind the widely-used Llama.cpp project has rolled out a significant technical update with commit b8878. This release, from the ggml-org team, focuses on expanding hardware acceleration support by adding a new 'DIAG' operation for Qualcomm's Hexagon DSP (Digital Signal Processor). The update includes HVX (Hexagon Vector eXtensions) support and DMA double buffering, which are critical for efficiently moving data and performing parallel computations on Snapdragon-powered mobile devices and edge hardware. This move directly challenges the dominance of Apple's Neural Engine and Nvidia's CUDA in the on-device AI space.
The commit, which also includes various fixes and performance optimizations, is part of a broader build that provides pre-compiled binaries for a vast array of platforms. These range from macOS on Apple Silicon and Intel, various Linux distributions (including with Vulkan, ROCm, and OpenVINO backends), to Windows with CUDA, Vulkan, SYCL, and HIP support. The addition of Qualcomm's ecosystem is a strategic play, tapping into the massive installed base of Android smartphones and IoT devices. For developers, this means they can now deploy the same Llama.cpp-optimized models across a wider spectrum of hardware, from data center GPUs down to power-efficient mobile processors, with fewer code changes.
- Adds Qualcomm Hexagon DSP support via new 'DIAG' op and HVX extensions for on-device AI.
- Enables ~30% faster inference for models like Llama 3 on Android devices with Snapdragon chips.
- Expands Llama.cpp's hardware reach beyond Apple/Nvidia to dominate the mobile and edge AI market.
Why It Matters
This democratizes high-performance, private AI by making powerful models run efficiently on the billions of Qualcomm-based devices in users' pockets.