Akashic's MemAttention boosts LLM agent accuracy by 10.2 points

Current LLM-based agents continuously accumulate context across multi-turn interactions, tool calls, and cross-session workflows. Replaying full history for each request quickly becomes impractical: long contexts increase prefill cost, may exceed context limits, and often bury task-relevant evidence in irrelevant content, degrading both serving efficiency and output quality. Researchers from multiple institutions propose Akashic, a low-overhead memory system built around MemAttention. MemAttention organizes context into bounded chunks and models semantic relationships across those chunks, preserving cross-chunk evidence without repeatedly rewriting the full history. Akashic further applies hardware-software co-designed memory placement to co-locate likely co-retrieved chunks, reducing retrieval fragmentation and I/O overhead.

Across four representative workloads (including multi-turn QA, tool use, and cross-session tasks) and three model sizes (7B, 13B, 70B), Akashic delivers consistent improvements over strong prior memory baselines: task accuracy up by 10.2 points, throughput up by 1.21x, and sustainable request rate up by 1.88x. These gains come without requiring expensive full-context recomputation. The paper's approach is particularly relevant for production agent systems where long-running sessions and complex tool chains are common. Akashic's chunk-based memory with hardware-aware placement offers a practical path to scale LLM agents to real-world workloads without sacrificing latency or accuracy.