New study exposes hidden privacy risks in long-lived AI agents' memory

A new study from researchers including Lei (Rachel) Chen, Guilin Zhang, Kai Zhao, and others (presented at ICML MemFM 2026) systematically investigates the privacy-utility trade-offs of long-lived AI agents' memory. Unlike prior work focused on static model weights or fixed memory configurations, this paper defines 'deployment-time memorization' as the emergent property of agents that remember users across interactions. The authors sweep three key memory-design knobs—summarization aggressiveness, retrieval breadth (k), and deletion mode—and evaluate them using two metrics: Personalization Recall (PR) and Adversarial Extraction Rate (AER), plus a new metric called Forgetting Residue Score (FRS) that quantifies how thoroughly deleted information can still be recovered from derived memory tiers. Experiments were conducted on the LongMemEval benchmark using Gemma 3 12B and GPT-4o-mini.

The results reveal a critical tension: aggressive key-fact summarization reduces canary extraction by 76% on Gemma and 64% on GPT-4o-mini, while preserving nearly all personalization recall. However, once content is compressed into summaries, increasing retrieval breadth (k) no longer restores leakage — a positive for privacy. But the same compression introduces a deletion-fidelity failure: when only raw data is deleted, derived summary copies remain recoverable in about 20% of instances. Only a full-pipeline purge (clearing both raw and derived memory) or tombstone redaction (marking items as deleted) drives worst-tier residue to zero. The paper argues that persistent agent memory must be evaluated as a first-class memorization mechanism — assessed by what it helps agents recall, what it makes extractable, and what it can truly erase. This has direct implications for compliance with data deletion regulations (e.g., GDPR) and for building trustworthy AI companions.