llama.cpp b9946 optimizes Hexagon DSP with tiled unary ops and faster inference
New release avoids VTCM overflow by tile-wide row processing and replaces divs with fastdiv on Hexagon.
llama.cpp's b9946 release focuses on Hexagon (Qualcomm DSP) optimizations: tiling for unary ops to prevent VTCM overflow, fastdiv replacement, host-computed kernel params, and specialized thread functions. The update also fixes std::min collision on Windows and enables LTO builds. The release includes builds for macOS Apple Silicon (arm64), macOS Intel (x64), iOS XCFramework, Linux (Ubuntu x64 CPU, ARM64 CPU, s390x CPU, x64 Vulkan, ARM64 Vulkan, x64 ROCm 7.2, x64 OpenVINO, x64 SYCL FP32/FP16), Android arm64, Windows (x64 CPU, ARM64 CPU, ARM64 OpenCL Adreno, x64 CUDA 12/13, x64 Vulkan, x64 OpenVINO, x64 SYCL, x64 HIP), and openEuler (x86 310p, x86 910b ACL Graph, aarch64 310p, aarch64 910b ACL Graph)—though macOS Apple Silicon with KleidiAI enabled and all openEuler variants are labeled DISABLED.
- Hexagon unary ops now use tiling to prevent VTCM overflow, improving reliability on memory-constrained DSPs.
- Division operations replaced with fastdiv, reducing instruction latency in kernel execution.
- Supports 10+ hardware backends including Apple Silicon, NVIDIA CUDA, AMD ROCm, Intel SYCL, and Qualcomm DSP.
Why It Matters
These optimizations make local LLM inference faster and more efficient on edge devices, enabling broader real-time AI applications.