New caching policy cuts multi-agent latency by 64.7% using workload signals

Multi-agent systems break complex tasks into directed acyclic graphs (DAGs) of specialized agent executions, creating natural opportunities to cache intermediate results across queries. However, existing cache eviction policies treat all cached entries uniformly based on simple access history, ignoring structural and workload signals unique to agentic environments. In a new arXiv preprint, Anas Mohamed and colleagues from the University of Waterloo and IBM Research introduce a workload-aware eviction policy that scores cached entries using three signals: recomputation cost (how expensive it is to regenerate), DAG dependency count (how many downstream agents rely on it), and agent invocation frequency (how often that agent type is called). These scores are fused into a single metric that retains the most valuable results under memory constraints.

Evaluated on three multi-agent benchmarks spanning diverse reuse patterns, the policy reduces end-to-end latency by up to 64.7% relative to an uncached baseline and achieves on average a 31.1% latency reduction over the next best finite-capacity baseline. Critically, it approaches the performance of an unbounded cache while maintaining accuracy on par with or exceeding all competing finite-capacity methods. The authors also demonstrate that their technique is complementary to plan-level caching (caching entire execution plans) and parallel agent execution, as each method targets a distinct efficiency bottleneck. This means teams deploying complex multi-agent pipelines can stack these optimizations for maximum throughput, making real-time AI workflows more practical than ever.