New Update Makes AI Chatbots on Your PC Run Faster

You may have used AI chatbots on websites, but there's another way: running them directly on your own computer. That's what llama.cpp does. It's free, open-source software that many people use to keep their conversations private or avoid monthly fees. This week, a new update (called b10730) made some of those local AI models noticeably quicker.

The update focuses on a family of models called Qwen. The developers found a clever way to streamline how the AI processes its "indexer heads" — a technical part of the model that scans your text. Instead of copying data around, the new code adds up the important pieces directly. In a test on a high-end graphics card, the speed of reading your prompts jumped from 2,170 to 2,366 tokens per second. "Tokens" are chunks of words, roughly 4 characters each, so this is a meaningful boost.

For you, the practical benefit is speed. If you're using a local AI to summarize a long report, analyze a big spreadsheet, or have a long chat session, you'll feel the difference. The faster the model reads your input, the sooner you get a reply. The developers say the speed gain grows even larger when the context window is long — that's the amount of text the AI remembers from earlier in your conversation.

Keep in mind this is a "pre-release" version, which means it's for testing, not for everyday users. The developers confirmed that the AI's output quality stays exactly the same — just faster. If you use an app built on llama.cpp, you might get this speedup after it updates. If not, you can wait for the stable release, but this gives a glimpse of how local AI keeps getting more practical.