New Trick Lets You Run Big AI Models on a Cheap Graphics Card
This could save you thousands on hardware and let you run powerful AI at home.
A developer has shared a clever trick that could let you run big AI models on a graphics card that's too small for them. Normally, a large language model like Gemma 4 needs a lot of video memory (VRAM) to run. If the model doesn't fit, you're out of luck. But this new approach splits the model: a compressed 2-bit version stays on the graphics card, while the rest of the model (6-8 bit) sits in your computer's regular memory (RAM). The graphics card quickly drafts the next word, and the full model checks it. This is called speculative decoding, and it's like having a fast intern make a guess and a senior editor verify it.
The developer measured that this method produces quality close to the full model while using much less memory. With a 6-bit residual, you get the same quality as an 8-bit model but read 28% fewer bytes from RAM. The draft also reuses the verifier's key-value cache, which means it accepts 25-30% more tokens per check than Google's official drafter. The CPU can check 16 tokens in about the same time as one because it's memory-bound.
The catch is that this only makes sense when your graphics card can't fit an 8-bit model. If it can, you're better off using existing software like llama.cpp. Also, the developer hasn't yet measured tokens per second, so real-world speed is unknown. But the idea is promising and could help people run powerful AI on cheaper hardware, saving money and making AI more accessible.
- A developer found a way to run large AI models on graphics cards with limited memory by splitting the model into two parts: a compressed version on the GPU and the full version in system RAM.
- This method cuts memory use in half and can be faster because the GPU drafts words and the CPU verifies them, with only small data transfers between them.
- It only helps if your graphics card can't fit the full model; otherwise, existing tools like llama.cpp are better.
Why It Matters
This could let you run advanced AI on a cheaper computer, saving money and making AI more accessible.