In-process memory cuts AI agent latency 83x, acts as extended working memory

A new paper from Yusuf Khan and Carlo Lipizzi, 'Memory in the Loop: In-Process Retrieval as Extended Working Memory for Language Agents,' challenges the conventional design where language agents query memory outside their observation-reason-act loop. The authors argue that latency, not the in-loop pattern itself, has been the barrier. By moving the store inside the agent process, retrieval times drop from tens to hundreds of milliseconds (network) to roughly 100 microseconds—three orders of magnitude faster. This speed enables the store to act as extended working memory rather than a tool consulted once per turn. Experiments with GPT-5-class models (GPT-5-nano, GPT-5-mini) under a bounded window showed that slower retrieval directly causes more redundant actions: 7.2 out of 12 steps were redundant at 110ms cloud latency vs. 0.0 at in-process speed (p=0.0079). Recall improved from 0/5 to 3.6–4.8/5 with in-loop memory, at p50 store operation times of 80–165μs. Notably, all 244 writes were retained across every run; every memory miss traced back to the agent's read policy, not the store. The paper also identifies network embedding as the dominant per-step cost (~200–400ms). Pairing the in-process store with a small local embedder brings total operation time to ~40μs, effectively eliminating the bottleneck. This work suggests a new paradigm for language agent architecture: treat memory as an intrinsic part of the reasoning loop, not an external API call.