P2P approach for LLM inference cuts latency with decentralized prefix caching

The paper tackles a critical bottleneck in LLM inference at scale: prefix caching reduces latency by reusing KV caches across requests with shared prompts, but cluster-scale reuse is limited because caches are partitioned across nodes. The authors propose a decentralized, prefix-cache-aware routing scheme for peer-to-peer (P2P) LLM serving. Each node maintains a local radix tree of its own cached prefixes and asynchronously refreshed estimates of peer caches using periodic anti-entropy (a gossip-style synchronization). Requests are routed to the node with the longest estimated prefix match, without any centralized coordination or expensive KV-cache transfer between nodes. A key insight is that stale metadata only causes cache misses (not incorrect outputs), making weak consistency sufficient for correctness—a practical trade-off for distributed systems.

Evaluation on simulated MMLU workloads reveals that decentralized routing improves latency under low communication delay and skewed prefix distributions (where many requests share similar prefixes). However, high network latency and affinity-induced hotspots limit benefits. The work demonstrates a promising avenue for scaling LLM inference without central servers, leaning on peer-to-peer principles. While still in simulation stage, it opens the door for more resilient, cost-effective inference in edge or volunteer computing environments. Future work could address hotspot mitigation and real-world deployment challenges.