DoYouRemember: New AI memory system beats VQ with 16x compression

Current multimodal LLMs (MLLMs) process images and then discard internal representations – they don't remember images. The paper "Do You Remember?" from arXiv highlights that while LLMs can understand images, their hidden states retain approximately zero recoverable visual information. The authors introduce DoYouRemember, a three-stage architecture: a VQ-VAE compresses images into discrete tokens, a LoRA-fine-tuned LLM attends to both visual and text tokens, and a Diffusion Decoder attempts to reconstruct images from hidden states. Experiments showed that the same decoder produces clear reconstructions from VQ-VAE tokens (pre-LLM) but pure noise from LLM hidden states (post-LLM), proving the LLM forgets.

The authors propose a shared memory matrix M that systematically fails under backpropagation due to gradient cancellation. They identify three root causes and solve them with local EMA updating – each image updates only its top-8 slots out of 64. The resulting M (229K parameters, 16x compressed) approaches the VQ upper bound. Scaling to 1,024 slots surpasses it (LPIPS 0.056 vs 0.071) because M's continuous representation avoids VQ quantization error. This work unifies memory as lossy compression, recall as decompression, and hallucination as inherent to lossy decompression – not a defect.