llama.cpp b9717’s K-Tails MMA Support Gives Power10 an Unexpected Speed Boost — Here’s Why It Matters

The llama.cpp project, the leading open-source C/C++ framework for running large language models locally, released version b9717 with a key optimization for IBM Power10 processors. The update focuses on the tinyBlas_Q0_PPC tiled matrix multiplication path, specifically for Q8 (8-bit quantized) and Q4 (4-bit quantized) kernels using Power10's MMA (Matrix Math Assist) hardware acceleration. Previously, the MMA matmul required the matrix dimension K to be divisible by kc (the panel size), causing many workloads to fall back to slower mnpack routines. With b9717, the code now supports K tails—processing the final, incomplete K panel using its actual depth and passing the reduced panel size through packing and kernel execution.

This change broadens the range of model sizes and configurations that can leverage Power10's native matrix accelerator, directly translating to faster token generation and lower latency for LLM inference on IBM Power servers. The fix was contributed by Aaron Teo (taronaeo) and is included in the standard build matrix that covers macOS (Apple Silicon and Intel), Linux (x64, arm64, s390x, with various backends like Vulkan, ROCm, OpenVINO, SYCL), Windows (x64, arm64, CUDA, HIP), and Android. For enterprise users running AI workloads on Power10 hardware—common in on-premises deployments—this optimization means more efficient use of existing infrastructure without requiring GPU upgrades. The release is tagged b9717 and is available now from GitHub.