b8914
New release boosts performance on Qualcomm Hexagon processors with vectorized operations.
The llama.cpp project, a popular open-source library for running large language models locally, has released version b8914. This update introduces a SOLVE_TRI operation specifically for Hexagon processors, which are digital signal processors found in many Qualcomm chips. The change, co-authored by Todor Boinovski, aims to optimize linear algebra computations critical for AI inference, particularly for models running on edge devices like smartphones.
The release also includes improvements to thread utilization, switching from chunk to batch processing for better efficiency on Hexagon hardware. Additionally, it vectorizes partial f32 loads and moves HVX (Hexagon Vector eXtensions) wrappers for f32 add, sub, and mul operations to a shared header file for cleaner code. The update is available across multiple platforms, including macOS (Apple Silicon and Intel), Linux (x64, arm64, s390x), Windows (x64, arm64), and Android (arm64), with support for various backends like Vulkan, ROCm, CUDA, and SYCL.
- Adds SOLVE_TRI operation for Hexagon DSPs, enhancing AI inference on Qualcomm hardware.
- Improves thread utilization with batch processing over chunk processing for Hexagon.
- Vectorizes partial f32 loads and moves HVX wrappers to a shared header for cleaner code.
Why It Matters
Optimizes local AI inference on mobile devices, enabling faster and more efficient LLM performance.