llama.cpp v9966 boosts inference speed with regex optimization
A single code tweak in llama.cpp slashes decode thread overhead by 30%.
The ggml-org team released llama.cpp version b9966 with a targeted performance fix that eliminates unnecessary regex recompilation. In tensor-split mode, the function `llama_meta_device_get_split_state()` runs once per tensor per token. Profiling revealed that rebuilding 29 std::regex objects on each call consumed most of the decode thread’s time — a classic hidden overhead. By marking the regex patterns as static const, they are now compiled just once and reused across all calls. Because static local variables are guaranteed to be initialized only once in a thread-safe manner (since C++11), the change is both safe and efficient.
The fix is minimal — just a few lines changed — but the impact is significant for users running large models across multiple GPUs with tensor splitting. Benchmarking showed the decode thread time dropped substantially, leading to faster token generation rates. No configuration changes are needed; upgrading to b9966 automatically applies the optimization. The release also includes builds for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Windows (CPU, CUDA, Vulkan, HIP), and Android. For local LLM enthusiasts and developers running inference at scale, this update translates to lower latency and higher throughput at no extra cost.
- 29 std::regex objects were being recompiled on every tensor call, dominating the decode thread.
- Fix: patterns marked static const, compiled once and thread-safe by C++11 guarantees.
- No user configuration needed — upgrade to b9966 for immediate inference speed improvements.
Why It Matters
Faster local LLM inference on multi-GPU setups means lower latency and higher token throughput for developers.