When Continual Learning Moves to Memory: A Study of Experience Reuse in LLM Agents

A new preprint from Nanyang Technological University (arXiv:2604.27003) challenges the assumption that memory-augmented LLM agents can sidestep the stability-plasticity dilemma by storing experiences externally. The authors, Qisheng Hu, Quanyu Long, and Wenya Wang, demonstrate that this challenge does not disappear but resurfaces at the memory level. Under a limited context window, old and new experiences compete during retrieval, effectively relocating the continual-learning bottleneck from parameter updates to memory access. To systematically study this, they propose a (k,v) framework that disentangles two fundamental design axes: how experience is represented (the key) and how it is organized for retrieval (the value).

Experiments in sequential-task settings using ALFWorld and BabyAI reveal several counterintuitive results. Abstract procedural memories (e.g., high-level task strategies) transfer more reliably across tasks than detailed trajectory logs. Negative transfer disproportionately harms hard cases—tasks that already struggle benefit less from shared memory and suffer more from interference. Most surprising: finer-grained memory organization is not universally beneficial. Designs that yield strong forward transfer can simultaneously induce severe forgetting, meaning there's a new trade-off at the memory level. The paper is still a work in progress but offers actionable insights for anyone building long-lived LLM agents that need to learn continuously without retraining.