b8578

The open-source project llama.cpp, maintained by ggml-org, has released a new update (commit b8578) focused on optimizing performance for Qualcomm's Hexagon digital signal processors (DSPs). The key fix addresses a "significant perf regression" of approximately 3-4 tokens per second (TPS) during text generation. This was traced to an incorrectly set 'in-order descriptor' bit in the DMA (Direct Memory Access) subsystem, which was unnecessarily serializing memory operations. The update also introduces a simple cache for mask rows to prevent redundant data fetches, further streamlining computation for AI models.

This technical optimization is crucial for developers and users running large language models (LLMs) like Meta's Llama 3 on mobile and edge devices powered by Qualcomm Snapdragon chips, which utilize Hexagon DSPs for efficient AI inference. By resolving this DMA bottleneck, the llama.cpp engine regains lost performance, making on-device AI applications more responsive and viable. The update is part of the project's continuous effort to support a wide range of hardware backends, as evidenced by its extensive pre-built binary support for macOS, iOS, Linux, Windows, and openEuler across CPU, GPU, and accelerator platforms.