Research reveals when distributed AI inference needs more bandwidth
AI inference bandwidth demand vs. GPU costs analyzed across 3-site testbed
Researcher Prasanna Caliaperoumal published a co-design evaluation examining when distributed AI inference requires increased wide-area bandwidth versus alternative approaches like KV recomputation or cache compression. The study introduces a workload model that predicts the crossover point where moving inference state across sites becomes more efficient than recomputing it—finding a context-independent threshold of 74-111 Gbps per stream for a 70B multi-head-attention model.
The paper quantifies five sensitivity axes including context length, attention architecture, queueing, agentic compounding, and bandwidth collapse due to loss/jitter. Economically, recomputation is cheaper at list GPU prices, but transfer becomes favorable when GPU scarcity and KV reuse inflate effective costs by 5-20x. Modern attention mechanisms shift the breakeven point by an order of magnitude in favor of data transfer. The findings position network levers—packet networks (millisecond-scale allocation) and optical fungibility (minute-scale capacity changes)—as economic complements rather than functional substitutes for overprovisioning.
- Crossover threshold for distributed AI inference vs. recomputation is 74-111 Gbps per stream for a 70B model (9-14 Gbps with grouped-query attention).
- Transfer becomes cost-effective when GPU scarcity and KV reuse inflate effective costs by 5-20x.
- Optical fungibility and packet networks offer economic advantages over functional overprovisioning in wide-area AI inference.
Why It Matters
Optimizing distributed AI inference bandwidth vs. compute costs can save enterprises 5-20x in operational expenses for large-scale deployments.