DMV-Bench tests AI agents' visual memory across 50-step shopping tasks
New benchmark reveals text-heavy agents fail at remembering what they saw in long tasks.
Most agent memory benchmarks focus on text, ignoring when an agent genuinely needs to remember what it saw. DMV-Bench, from researchers Yujin Tang, Chenming Shang, Ruize Xu, and Nikhil Singh, addresses this gap. It simulates a home-furnishing e-commerce environment with 1,000 product variants. Each session presents a chain of shopping steps (from 5 to 50), where every visited product image contains a unique pre-rendered incidental cue. A 'text-leakage contract' ensures the discriminative signal stays in the pixels—agents cannot rely on captions or textual metadata.
To solve DMV-Bench, the team proposes DualMem, a memory architecture inspired by dual-coding theory. It maintains parallel visual and verbal codes: the visual channel carries the cue end-to-end, while the verbal channel aids query grounding. DualMem outperforms a caption-only baseline and three recent multimodal memory systems on both Gemini 2.5 Flash and Qwen2.5-VL-7B across all chain lengths (5, 10, 15, 50). The advantage persists even when controlling for memory-bank size and encoding-position bias. This work highlights a critical blind spot in current AI agent design—visual episodic memory—and offers both a rigorous evaluation tool and a practical solution.
- First interactive benchmark (DMV-Bench) specifically for multimodal agents' visual memory, using 1,000 product variants with pixel-only cues.
- DualMem architecture beats three existing systems on Gemini 2.5 Flash and Qwen2.5-VL-7B across task chains of 5, 10, 15, and 50 steps.
- Uses a 'text-leakage contract' to ensure agents rely on visual memory, not textual shortcuts.
- DualMem's asymmetric coding keeps visual channel for cues, verbal channel for grounding; outperforms caption baselines.
Why It Matters
As AI agents go multi-step, visual memory is the missing link—DMV-Bench and DualMem pave the way.