Memorization In Stable Diffusion Is Unexpectedly Driven by CLIP Embeddings
A single token swap can stop Stable Diffusion from regurgitating training images.
A CVPR 2026 paper by Bumjun Kim and Albert No reveals that memorization in Stable Diffusion isn't random—it's driven by an unexpected property of CLIP text embeddings. The authors categorize input tokens into <startoftext>, <prompt>, <endoftext>, and <pad>, and find that <prompt> embeddings contribute minimally in memorized cases. Instead, the <pad> token—typically used for padding sequences to uniform length—structurally duplicates the <endoftext> embedding (v^eot), which is the only embedding explicitly optimized during CLIP training. This duplication unintentionally amplifies v^eot's influence, causing the model to over-rely on it and thereby driving memorization of specific training images.
The team proposes two simple, inference-time mitigation strategies that don't require prior detection of memorized prompts. First, replace the tokenizer's default <pad> from the <eot> token to the '!' character before embedding, and mask the v^eot embedding entirely. Second, partially mask the <pad> embeddings to reduce their influence without completely removing them. Both methods effectively suppress memorization without degrading image quality, and they can be dropped into existing Stable Diffusion pipelines as a post-hoc fix. The paper is accepted at CVPR 2026 Findings, and the code is open-sourced, making it immediately actionable for developers concerned about data leakage and copyright issues in generative models.
- <pad> token's embedding duplicates the structure of <endoftext>, amplifying its influence and causing over-reliance on a single optimized embedding.
- Two simple inference-time fixes: replace <pad> with '!' token and mask v^eot, or partially mask <pad> embeddings — both suppress memorization.
- Mitigation works without prior detection or retraining, and does not degrade image quality (accepted at CVPR 2026).
Why It Matters
A plug-and-play fix for Stable Diffusion memorization that protects privacy without sacrificing output quality or requiring model retraining.