llama.cpp b10078 refactors Vulkan queues for faster LLM inference
New Vulkan update bypasses mutex locks when driver supports internal synchronization, boosting throughput.
llama.cpp, the popular open-source library for running large language models locally, has released version b10078 with a significant Vulkan backend overhaul. The core change refactors the vk_queue system to use per-instance mutexes and unique handles, integrating the VK_KHR_internally_synchronized_queues extension. This allows the driver—rather than the CPU—to handle synchronization when supported, bypassing expensive host-side mutex locking. The refactor abstracts queue submission into a polymorphic interface that detects driver capabilities at runtime, falling back to traditional mutexes when needed.
Beyond the performance gains, this release includes several fixes: compilation errors on various platforms, duplicate pNext chains for feature queries, and proper null checks in destructors. The commit, contributed with guidance from NVIDIA’s Jeff Bolz, also streamlines internally synchronized queue detection to avoid redundant API calls. For users, this means smoother multi-threaded execution and lower latency when running models like LLaMA, Mistral, or others via llama.cpp on Vulkan-compatible GPUs.
- Integrates VK_KHR_internally_synchronized_queues to bypass CPU-side mutexes when driver supports it.
- Refactors queue submission into a polymorphic interface with per-instance mutexes and unique pointers.
- Fixes multiple compilation errors, pNext chain issues, and device creation timing for Vulkan backends.
Why It Matters
Lower overhead and better parallelism for local LLMs on Vulkan GPUs means faster inference without NVIDIA CUDA.