Latent Cache Flow shrinks AI agent communication 96%, speeds 8.5x

Latent Cache Flow (LCF), introduced by researchers Rossi, Raghunath, and Wu, tackles a critical bottleneck in multi-agent LLM systems: communication latency and information loss. Current agents must decode and re-encode text autoregressively, wasting tokens and time. Prior work like Cache-to-Cache (C2C) learned large adapters to exchange KV caches, but required identical contexts and used massive 956 MB adapters. LCF compresses and jointly translates keys and values, slashing adapter size to just 13 MB—about 4% of C2C's. It also handles differing contexts by transmitting a summary of new information, enabling agents with different conversation histories to communicate efficiently.

In experiments, LCF's 13 MB adapter outperformed C2C's 956 MB adapter in shared-context settings on accuracy, and in different-context scenarios it achieved 23% higher accuracy and 8.5x faster throughput compared to text-based communication. This breakthrough could dramatically accelerate collaborative AI tasks like multi-step reasoning, code generation, and research agents that currently rely on slow text exchanges. By shrinking the communication overhead to a lightweight cache flow, LCF opens the door to real-time agent swarms operating at scale with minimal latency.