Memory Dial: A Training Framework for Controllable Memorization in Language Models
New training method lets scientists dial memorization pressure up or down with a single parameter α.
Researchers Xiangbo Zhang and Ali Emami have introduced Memory Dial, a groundbreaking training framework that treats memorization in language models as an explicit, tunable variable rather than an opaque byproduct. The core innovation is a single parameter α that interpolates between standard cross-entropy loss and a temperature-sharpened objective. This creates a family of models—identical in architecture, data, and optimization—that differ only in their 'memorization pressure.' Experiments across six model architectures and five benchmarks confirmed that α reliably controls this pressure: accuracy on seen training examples increases monotonically with α, while performance on unseen data remains stable.
Memory Dial's experiments revealed three key findings. First, larger models are significantly more responsive to changes in memorization pressure. Second, frequent sequences in the training data are memorized more easily than rare ones. Third, the effect is robust across different sharpening temperatures and even transfers to multilingual settings. This framework moves beyond post-hoc detection methods, allowing researchers to systematically study how memorization emerges and interacts with a model's ability to generalize. Accepted to ACL Findings 2026, Memory Dial provides the research community with a precise experimental tool to disentangle the complex relationship between memorization and generalization, which is crucial for improving model interpretability and safety.
- Introduces a single parameter α that directly controls memorization pressure during model training, creating otherwise identical models for comparison.
- Validated across six architectures and five benchmarks, showing α reliably increases memorization of seen data without harming generalization on unseen tasks.
- Reveals that larger models are more responsive to memorization pressure and that frequent sequences are memorized more easily than rare ones.
Why It Matters
Provides researchers with a controlled method to study and potentially mitigate unwanted memorization, a key concern for model safety and copyright.