MemArena benchmark tests on-device AI memory assistants

Researchers have introduced **MemArena**, a new benchmark designed to rigorously test on-device AI agents specializing in personal memory assistance. Published on arXiv, the work is authored by Jiadong Zhang and Xiaosong Ma, and aims to address gaps in existing benchmarks by simulating realistic, ego-centric interactions over extended periods. The benchmark uses the MASim agent simulator to replicate 50 agents over 15 days, generating 10.3 million dialog text tokens and 24.1K ego-observed tokens per agent each day. This setup enables evaluation across six dimensions: recall, reasoning, and trustworthiness.

The study evaluates five open-weight memory backends—Vanilla context, BM25-RAG, Oracle retrieval, Memobase, and MemSearch—paired with Qwen3-0.6B readers. One key finding is that the choice of memory backend significantly impacts content accuracy more than scaling the reader model. For instance, switching from Memobase to MemSearch delivered a **+32.5/+19.2 percentage point improvement** in accuracy metrics, surpassing gains from reader scaling alone (+10.6/+6.8 pp). However, the research also uncovered systemic issues: permission-aware access mechanisms failed across all tested backends, with the Oracle backend leaking data excessively while others were overly restrictive. Latency tests on a Spark GB10 edge node showed memory-search adds only 87/7/48 ms (BM25-RAG/Memobase/MemSearch), indicating minimal impact on total time-to-first-token (TTFT) for most configurations.