llama.cpp, the Free Way to Run AI Offline, Just Got a Fix

Llama.cpp is a piece of free, open-source software that has been downloaded and starred hundreds of thousands of times. Its whole purpose is unusual in the AI world: instead of sending your questions to a company's servers, it lets you run an AI model directly on your own computer, phone, or a cheap mini-PC. That means no monthly bill, no internet connection needed, and nothing you type leaves your device.

The update released this week, labelled b10896, is not a headline-grabbing new feature. It fixes a specific bug. When the software tried to run AI models that understand both pictures and words — what insiders call "vision models" — a speed-boosting trick called speculative decoding could crash. Speculative decoding is best thought of as a guess-and-check shortcut: the AI drafts several words at once, then verifies them, and it makes responses come out noticeably faster. The bug meant the drafting part ran out of memory when images were involved, so the whole thing fell over. The fix tells the software to stop copying image data around unnecessarily.

The release page also lists dozens of ready-made versions for different hardware: Macs with Apple chips, Windows PCs with Nvidia graphics cards, Android phones, and even some unusual server chips from China. That breadth is the point. A person with an ordinary gaming laptop and a person with a data-centre machine can both use the same free tool, which is why it has become a quiet standard in the do-it-yourself AI community.

So what's the practical takeaway for you? Very little changes today unless you already run local AI with images. But the bigger trend is worth watching: the ability to run a capable AI assistant privately, offline, and for free keeps improving month by month. Small fixes like this one are the incremental steps that get us there.