Developer Tools

llama.cpp b9969 fixes long promts on Adreno GPUs

New release resolves shared memory bug for q4_0 quantized networks on mobile

Deep Dive

The llama.cpp team released b9969, a critical update for the popular local LLM inference engine. The headline fix resolves a Vulkan-specific issue where the llama-cli would break when processing longer prompt sizes with q4_0 quantized networks. The root cause was traced to insufficient shared memory on Adreno GPUs, common in mobile devices like Qualcomm Snapdragon-based phones and tablets. The update includes a patch to route large matrix multiplications to medium tile sizes, avoiding the shared memory bottleneck.

Beyond the Vulkan fix, b9969 cleans up unused Adreno device entries and optimizes the matmul pipeline for small workloads. The release package spans all major platforms: macOS (Apple Silicon with optional KleidiAI, Intel x64), Linux (x64, arm64, s390x, plus Vulkan, ROCm 7.2, OpenVINO, SYCL), Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and Android arm64. The fix is particularly important for developers running local LLMs on mobile hardware, where Adreno GPUs are prevalent.

Key Points
  • Fixes llama-cli crash for long prompts on Adreno GPUs running q4_0 quantized models
  • Root cause: insufficient shared memory; resolved by routing large matmuls to medium tile
  • Includes platform builds for macOS, Linux, Windows, Android, and iOS XCFramework

Why It Matters

Local LLM inference on mobile GPUs becomes more reliable, expanding edge AI capabilities.

📬 Get the top 10 AI stories daily