A Free AI Chatbot Now Runs on an Ordinary Gaming PC

What happened: A hobbyist posted a thank-you note to a volunteer known online as Mradermacher, who spends his time 'quantizing' AI models. Quantizing means shrinking a giant AI's brain so it fits on smaller, cheaper computer chips. Using that shrunken version, the hobbyist got a large open-source chatbot (roughly 26 billion settings, a version of Google's Gemma) running on two Nvidia RTX 4060 cards — mid-range graphics cards that cost a few hundred dollars each, not the $30,000 data-center gear this used to require. It generated 75 tokens per second, where a 'token' is a small chunk of a word.

Why that's a bigger deal than it sounds: 75 tokens a second is about 55 words a second — far faster than anyone reads. So a model living on your own desk answers instantly, with no waiting on the internet. Nothing you type is sent to a company's servers. There's no monthly bill, no usage limit, and no chance your private documents end up training someone else's product. The software used, LM Studio, is a free app that lets non-experts download and chat with these models, and it was serving answers to another tool called Hermes.

Now the honest catch. This is still hobbyist territory. It took two graphics cards, careful setup, and someone who knows what a 'quant' is. The shrunken model is slightly dumber than the full version, and noticeably weaker than the best paid chatbots from OpenAI or Google. It's also a Reddit post, not a product announcement — one person's setup, not a guarantee you'll get the same result. Expect heat, fan noise, and a good few hours of fiddling if you try it yourself.

The bigger story is the trend. A quiet army of volunteers like Mradermacher does the unglamorous compression work that makes open AI models run on everyday machines. Every few months the models get better and the hardware bar gets lower. Within a couple of years, 'your AI assistant lives in your PC' could be as ordinary as having a printer — and just as private.