AgentMemBench: External key-value memory crushes context windows for AI agents
AI agents forget after long chats—this benchmark tested 5 memory fixes and found a clear winner.
AgentMemBench, a new benchmark from researcher Ahmed Cherif, systematically compares five long-term memory management strategies for conversational AI agents: in-context windowing (ICW), external key-value stores (EKV), graph-based episodic memory (GEM), compression-based summarisation (CBS), and web-augmented memory (WAM). Using Qwen2.5-7B-Instruct (4-bit) for both generation and judging, the benchmark spans three datasets—LoCoMo, MultiDoc2Dial, and MSC—and measures Recall@k, MRR, nDCG@k, Answer F1, Faithfulness, Memory Footprint, and Latency across 491 annotated question turns.
Results show EKV wins on every quality axis: macro Recall@5 of 0.792, MRR 0.677, and F1 0.156. On LoCoMo, where the answer lies many sessions back, ICW, WAM, GEM, and CBS retrieve almost nothing (Recall@5 ≤ 0.005), while EKV alone reaches 0.573—demonstrating that recency windows, summaries, and entity graphs collapse at long horizons. CBS is the runner-up at 0.556, but EKV's recall advantage comes at a cost: ~5,100 tokens per turn vs ~300 for ICW/WAM. The paper also evaluates MemGPT/Letta and HippoRAG under the same harness, and releases full code and results for reproducibility.
- External key-value stores (EKV) hit 0.792 Recall@5, beating all rival memory strategies across quality metrics.
- On LoCoMo's long-range recall task, EKV scores 0.573 while context windowing, WAM, GEM, and CBS all score below 0.005.
- EKV's accuracy costs ~5,100 tokens vs ~300 for windowing, making the accuracy-efficiency trade-off explicit.
Why It Matters
Memory architecture decides whether AI agents can actually remember across sessions—dense retrieval beats shortcuts.