Fathom's Trick Lets AI Read Less Memory and Answer Faster

When you have a long chat with an AI assistant — especially one that's doing tasks for you, like writing code — the AI keeps notes on everything said so far. That pile of notes is called the KV cache. After a million words of conversation, that pile is enormous. The problem: to answer your next question, the AI has to dig through the whole pile, and that digging, not the thinking, becomes the slow part.

Fathom attacks that digging. Normally an AI reads a fixed amount of detail from every stored note — like photocopying 136 pages of a book every time you need one fact. Fathom instead lets each question decide how much detail it needs, and where. It stores the notes in layers, like a photo saved at different resolutions, so the AI can read the blurry version for easy questions and the sharp version only when precision matters. A clever formula decides where to spend its limited reading budget.

The numbers are decent, not miraculous. On a common open model called Qwen3-8B, with a million words of context, Fathom was 1.67 times faster than the best competing methods. It also read 18 percent fewer bytes of data while making fewer mistakes, in six of seven test setups. On realistic coding-assistant sessions, it matched the most accurate method's answers while reading 92 bits instead of 136.

The catch: this only helps when the AI's memory is offloaded to ordinary computer memory rather than sitting on the expensive graphics chip. If everything already fits on the chip, Fathom does nothing. It's also a single-author research paper, not a product — so expect it to show up inside AI services rather than something you download. Still, the direction is clear: less hardware, longer conversations, lower bills.