Free AI App llama.cpp Just Got Up to 118x Faster on Some PCs

llama.cpp is a free piece of software that lets people run AI chatbots directly on their own laptop or desktop — no subscription, no internet, no company reading your chats. The update, labelled b10956, rewrites one small but constant chore the AI does over and over: picking the best few options from a giant list of possibilities. Think of it like choosing your top 3 restaurants out of 200,000, thousands of times per second. On Intel graphics chips, the old code simply gave up when the list got long and handed the job to the main processor — a slow round trip every single time.

The new approach, called "radix select," does the whole job on the graphics chip instead. It's like sorting mail into buckets rather than comparing every letter to every other letter. The payoff: one heavy test case dropped from about 358 microseconds to roughly 15x–20x faster territory, with the developers reporting 4.98x to 118x improvements depending on the task. The 118x figure is the flashy one; the everyday gain is closer to 5x. That's the difference between an AI that thinks for a second before answering and one that feels like typing to a friend.

The catch is who actually benefits. This work targets Intel graphics hardware through a system called SYCL — a way of running AI code on Intel chips. If your computer has an Nvidia card or is a Mac, this update does nothing for you today. It's also a pre-release build, not a polished product, so it may be rough around the edges. And in a few narrow situations, the old slower method is actually faster, so the developers deliberately kept both.

Still, this is a small window into a bigger trend. Running AI at home keeps getting cheaper, faster and more private. Every speed bump like this makes it more realistic to ditch the cloud subscription and keep your data on your own machine.