Memory-R2 framework fixes unfair credit assignment in long-horizon LLM agents

Memory-augmented LLM agents can store, update, and reuse information across sessions, but training them with reinforcement learning is tricky: once different rollouts write different memories, they no longer share the same environment. This violates the key assumption behind group-relative methods like GRPO, where rollouts are compared as if sampled from the same state, leading to noisy or biased credit signals for memory operations.

To solve this, researchers from LMU Munich, Siemens, and elsewhere introduce Memory-R2 with a novel core algorithm called LoGo-GRPO. LoGo-GRPO combines local group-relative optimization—rerolling different memory operations from the same intermediate memory state for fairer comparisons—with global trajectory-level rewards that preserve end-to-end learning. The framework also uses a shared-parameter co-learning design: a fact extractor and memory manager are instantiated from the same LLM backbone using role-specific prompts, jointly optimizing memory formation and evolution. To stabilize multi-step RL over long horizons, Memory-R2 employs a progressive curriculum that gradually increases training from 8 to 16 to 32 sessions.