llama.cpp release b9965 boosts sorting on Hexagon DSP chips
New release speeds up tensor sorting by 10x on Qualcomm Hexagon processors
The llama.cpp project (ggml-org) has released version b9965 with a focus on improving the ARGSORT operation for small tensors on Hexagon DSP architectures. The update adds a highly optimized bitonic sort implementation that runs entirely in HVX vector registers, handling up to 1024 elements without spilling to memory. The code specializes sort functions for common cases, adds tracing and local context support, and fixes inverted vrors (vector register rotate) in the hex-sort module.
This optimization is particularly impactful for running large language models on edge devices equipped with Qualcomm Hexagon digital signal processors (DSPs). Small tensor sorting frequently appears in attention mechanisms and token sampling routines. By making ARGSORT up to 10x faster in these scenarios, the release directly reduces inference latency on smartphones, IoT devices, and other embedded systems running llama.cpp locally, bringing desktop-quality AI performance to mobile hardware.
- Optimizes ARGSORT for small tensors on Hexagon DSP using a bitonic sort approach
- Handles up to 1024 elements entirely in HVX registers for maximum speed
- Includes specialized sort functions and fixes for inverted vrors operations
Why It Matters
Crucial for running large language models efficiently on Qualcomm-powered edge devices, reducing inference latency significantly.