This Free Tool Runs AI on Your PC, and It Just Got Faster

Most people use AI through websites like ChatGPT, which send your questions to giant cloud servers. But there's a growing alternative: llama.cpp, a free open-source program that lets you run AI models directly on your own laptop, desktop, or phone. This means no subscriptions, no data leaving your device, and no internet needed. The project has over 126,000 stars on GitHub, making it one of the most popular ways to run local AI.

The latest update, version b10657, makes these local models run faster on graphics cards (GPUs). It adds special helper programs, called kernels, that let the card handle a clever AI trick known as "mixture of experts." In plain terms, instead of one massive brain working on every task, this type of model splits work among specialized mini-experts. That makes it smarter and more efficient, but trickier to run. The new kernels help GPUs juggle these experts more smoothly, so answers come back quicker and use less memory.

The update works across Windows, Mac, Linux, and Android devices, so most people can benefit. For everyday users, the biggest wins are privacy and cost. You can chat with a capable AI assistant completely offline, and no one else sees your conversations. You also avoid monthly cloud fees and can keep using your hardware for as long as it works.

That said, this is a pre-release version, meaning it's not fully polished yet. Installing it requires downloading files and possibly using a command line, which might scare off non-technical folks. If you're not comfortable with that, you might wait for the stable release. But if you want to try cutting-edge local AI, this is a clear step forward.