llama.cpp b10194 adds CUDA transpose-free gemmv for faster inference

ggml-org has released llama.cpp b10194, the latest update to the popular C++ library for running LLMs locally. The headline change is a CUDA optimization that allows transpose-free gemmv (general matrix-vector multiply) computation. When a weight matrix is shaped 1xK, the new code path skips the usual transpose and directly uses mat_mul_vec_f. This reduces memory overhead and kernel launch complexity, which can translate into faster inference on NVIDIA GPUs. The release is tagged b10194 and includes PR #26171, signed by GitHub's verified signature. It's part of the ongoing effort to squeeze more performance out of consumer and datacenter GPUs.

The update ships with pre-built binaries for a wide range of platforms: macOS (Apple Silicon with optional KleidiAI, Intel), iOS, Linux (Ubuntu x64/arm64, s390x) supporting CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL FP32/FP16, plus Android arm64 and Windows (x64/arm64) with CUDA 12.4/13.3 DLLs, Vulkan, OpenVINO, and HIP. This ensures developers can easily test the new optimization. While the release doesn't provide benchmark numbers, the transpose-free approach is particularly beneficial for small weight matrices and single-layer operations, common in token generation. For developers running models like Llama or Mistral via llama.cpp, this update promises better GPU utilization and lower latency, making local AI inference more practical for real-time applications.