b8121
New commit delays CUDA graph activation until warmup completes, avoiding wasted overhead on volatile graphs.
The ggml-org team behind the widely-used llama.cpp project released commit b8121, a significant optimization to their CUDA backend. The update addresses suboptimal CUDA graph capture behavior where graphs were eagerly enabled on first compute call, then permanently disabled after 4+ consecutive property changes. The new approach introduces a warmup period, delaying CUDA graph activation until the same graph is called at least twice with matching properties. This prevents wasted capture overhead during volatile graph phases (like prompt processing) and allows graphs to re-enable once they stabilize (during token decoding). The fix specifically addresses issues discussed in GitHub thread #19708 and was co-authored by Johannes Gäßler and Aman Gupta. Llama.cpp powers local AI inference for 95.5k GitHub stars and this optimization benefits users running models like Llama 3 on NVIDIA hardware.
- Delays CUDA graph activation until warmup completes (same graph called twice with matching properties)
- Prevents wasted capture overhead on volatile graphs and allows re-enabling after stabilization
- Fixes GitHub issue #19708 where graphs were permanently disabled after property changes
Why It Matters
Faster, more efficient local AI inference for developers running models like Llama 3 on consumer NVIDIA GPUs, potentially doubling stable workload speed.