A 68GB AI Model Now Runs on a Cheap Gaming PC

Two months ago, a developer wondered if predicting which parts of a large AI model would be used next could speed up running it on a normal computer. They shelved the idea because their first attempt cheated by skipping some parts, which hurt the AI's answers. Recently, a new AI model called Qwen 3.8 Flash Next came out—it's 68GB, way too big for most home PCs. The developer decided to try again, but this time without cutting any corners.

Their setup: an RTX 3060 graphics card with 12GB of memory, 16GB of regular RAM, and a fast SSD. Normally, running such a big model on this hardware is painfully slow—about 1.4 to 2.1 words per second—because the computer constantly has to fetch data from the SSD. Some other tools claim 40 words per second, but they require at least 24GB of RAM to hold parts of the model in memory. With only 16GB, you're out of luck.

So the developer added a new feature to the free software llama.cpp called --moe-direct-io. It doesn't drop any parts of the model, so the output is exactly the same as the original. Instead, it predicts which parts will be needed next and loads them from the SSD in advance. The result: 20 to 21 words per second, with almost no slowdown from disk reads. That's about 15 times faster than the standard method, and it works on a budget PC.

The catch? You need a fast SSD and some technical know-how to set it up. But this shows that with clever software, even huge AI models can run on everyday hardware. That could mean more privacy, no monthly fees, and AI that works offline.