llama.cpp b10171 fixes massive perplexity bug on Adreno GPUs
New release slashes perplexity from 1817 to 15.6 on Adreno 740
llama.cpp, the popular open-source LLM inference engine by ggml-org, has released version b10171 with a critical bug fix for OpenCL Adreno GPUs. The issue affected the KQ/KQV image kernels (ggml_cl_mul_mat_kq_kqv_adreno), which ignored dimension 3 (ne03/ne13) when handling multi-stream batches. In practice, this meant that for any batch with more than one sequence (e.g., llama-perplexity with -b 2048 and n_seq=4, or multi-slot llama-server queries), the kernel only read the first stream's key/value data, producing garbage for all subsequent streams. Flash attention could mask the bug when enabled, but on devices where FA is declined (like Adreno 740), default settings triggered the error, causing perplexity values as high as 1817-1944 instead of the expected ~15.6.
The fix routes tensors with ne03/ne13 > 1 to a general path that correctly handles the fourth dimension, and also honors view_offs when creating sub-buffers (previously always 0 but could cause misreads with future views). Testing on Adreno 740 with Llama-3.2-1B-Instruct Q4_0 and wiki.test.raw showed perplexity dropping from 1817.64 to 15.61 (default) and 1941.64 to 15.61 (with -fa 0). Adreno 840 saw a similar improvement from 1943.90 to 15.50. Single-stream results remained unchanged (15.6090). The release also includes various build options for macOS, Linux, Windows, Android, and more. This is a must-update for anyone running multi-sequence inference on Qualcomm Adreno GPUs.
- Bug in OpenCL Adreno KQ/KQV kernels ignored tensor dimension 3, causing garbage output for multi-stream batches (e.g., -b 2048 with n_seq=4).
- Perplexity dropped from ~1817-1944 to ~15.6 on Adreno 740 after the fix; single-stream results unaffected.
- Fix routes ne03/ne13 > 1 to general path and honors view_offs for sub-buffers; also improves robustness for future tensor views.
Why It Matters
Critical patch for Adreno GPU users needing correct multi-sequence LLM inference on mobile, desktop, or edge devices.