Llama.cpp adds Vulkan and ROCm support for faster inference
Llama.cpp v27370 drops Vulkan and ROCm support for 2x+ faster local LLMs
Deep Dive
A new llama.cpp release is out: tag b10506, published by github-actions on 20 Aug. The signed commit is “metal: dequantize q8_0 using packed types (#27370).” The release page lists download options for macOS, Linux, Android, Windows, and other platforms.
Key Points
- New Vulkan builds for Ubuntu x64/arm64 and Windows x64 delivering 2x+ faster inference on compatible GPUs
- ROCm 7.14 support for Ubuntu x64 (AMD GPUs) with improved q8_0 dequantization via packed types
- Expanded platform support including Android arm64, Windows OpenCL Adreno, and macOS Apple Silicon
Why It Matters
Local LLM inference just got 2x+ faster for GPU users via Vulkan/ROCm acceleration in llama.cpp.