Your AI assistant can recall your entire chat history. But when it's time to book that flight, it might still forget to use your frequent flyer number. That's the brutal finding from a new benchmark called MERIT, posted just days ago on arXiv. Researchers built a test to see if memory actually helps AI agents that use tools—and the results are embarrassing for anyone selling "perfect recall" as the next big thing.
Here's the setup: MERIT measures the marginal utility of memory for task performance, not just whether a model can answer questions about dialogue history. The researchers point out that today's memory evaluations—like LoCoMo and LongMemEval—test whether a model can answer questions about past conversations. But that's like grading a chef on whether they remember your order, not whether they actually cook the meal. MERIT asks: does remembering past tasks change what a tool-using agent does?
The answer: it depends. On tasks where one step depends on another, memory boosts success from near zero to as high as 100%. That's a massive swing—imagine going from a coin flip to a sure thing. But here's the twist: full memory replay is never economical. Never. You can't just dump the entire history into the context window and expect efficiency. And even when the AI retrieves the right fact, it acts on it only 55% of the time. That's barely better than a coin toss. So the agent knows your frequent flyer number, but half the time it ignores it anyway.
This isn't just an academic quibble. It's a gut punch to the entire "memory as a feature" narrative. Companies are racing to build AI agents that remember everything—your emails, your calendar, your last 50 conversations. But if those agents only act on that memory half the time, what's the point? You're paying for a goldfish with a PhD.
The related research makes this even more interesting. Another paper on subagents vs. agent skills compares two ways to handle long-horizon tasks: cramming every instruction into the context window versus executing reusable knowledge. That's the same tension MERIT exposes. Memory isn't free—it's a cost. And like any cost, you need to weigh it against the benefit. The MERIT authors aren't saying memory is useless. They're saying we've been measuring the wrong thing. Recall isn't action. Knowing isn't doing.
So what's next? Trends suggest the industry could pivot toward cost-aware memory systems that prioritize which facts to retrieve based on the task at hand. Maybe we'll see agents that learn to forget—strategically. Or maybe the 55% action rate becomes the new benchmark to beat. Either way, the era of "more memory = better AI" is over. The question now: can we build agents that don't just remember, but actually use what they remember? The race is on.