Free Local AI Runs 2x Faster Thanks to New llama.cpp Update

llama.cpp is a free, open-source program that lets anyone run AI models directly on their own computer instead of using cloud services like ChatGPT. That means your data never leaves your machine, there are no monthly fees, and everything works even without internet. The catch? It's been a bit slow on regular hardware.

This new pre-release version, b10731, fixes a bottleneck that wasted time during text generation. When AI writes, it predicts several words at once, then checks them. If the prediction is wrong, it has to "roll back" to a previous thought — and the old way of doing that forced the computer to save its entire memory every time. The update uses quick snapshots instead, dramatically cutting that overhead.

Real-world numbers on one advanced model (Qwen4-exp) show the gain: prose speed jumps from 83 tokens per second to 144, and code from 123 to 183. Tokens are just chunks of text — a token is roughly half a word. So a response that used to take 10 seconds now takes about 6, right on your own hardware.

This is a big step toward making local AI feel snappy enough for daily use. There is a catch: this is an early developer release, so the average user won't see it immediately. But eventually it'll roll into beginner-friendly apps, and every speed boost makes private, free AI more practical for everyone.