AI's Working Memory, Not Size, Is What Chokes Your Computer

When people run AI on their own computers, they usually worry about one number: how big the model is, or its "brain size." The thinking goes that the bigger the parameters, the more powerful the AI, and the more graphics card memory you need. But there's a hidden problem that gets far less attention: the AI's working memory, known as a KV cache. Think of it like a student taking a test: they need to keep every fact they've read fresh in their head to answer questions correctly.

Every new word you send to AI, or every new word it generates, gets added to this working memory. If you're having a long conversation or asking the AI to process a 200-page book, that memory keeps growing. The result? A model that fit fine on your graphics card at first can suddenly run out of room at 100,000 or 200,000 words of context. This is why typing a long chat or dropping a massive document into a local AI can make it incredibly slow—or just crash.

There are some clever tricks already helping. GQA and MQA are ways to let different parts of the AI share memory instead of each keeping a separate copy. Quantization shrinks the numbers used in memory, like compressing a file. But no matter what, the memory still grows with every additional word of context. It's like trying to save a text that gets bigger every second—you can compress it, but it keeps expanding.

This changes what "better local AI" will mean going forward. Instead of just building smaller and smaller models, engineers will likely focus on making AI that knows how to forget or summarize old information when it's no longer needed. The future question might not be "How smart is this AI?" but "How much does it need to remember?" For everyday users, that means your personal AI might soon run smoothly on your laptop—just not by getting tiny, but by getting more selective about what it holds in its head.