NVIDIA Dynamo study reveals 3.1x efficiency gain with adaptive routing

A new paper from Athos Georgiou (NCA) provides the first formal game-theoretic analysis of disaggregated inference architectures, using NVIDIA Dynamo as a case study. The architecture physically separates prefill and decode phases onto distinct GPU pools, creating competing 'agents' that share a fixed hardware budget. The researchers model the system as three coupled games: a two-player resource game between pools, a selfish caching game over the hierarchical KV cache, and a congestion game with positive externalities. They empirically validated the latter two on a 3-node NVIDIA B200 cluster running Dynamo v0.9.0 with Nemotron-4-340B (TP=8) and Llama-3.1-70B (TP=4).

Key findings include a three-regime Price of Anarchy (PoA) structure that shifts when GPU saturation occurs. Below saturation, selfish behavior has bounded PoA; at saturation, superlinear latency and cache externalities drive PoA upward. The authors designed an adaptive controller that detects saturation in real time and adjusts routing—switching from cache-affinity exploitation to load-balanced congestion avoidance. On the 70B 1P/5D topology, PoA dropped 3.1x (from 66.4 to 21.5) in the saturated phase at a 13% throughput cost. On the 70B 1P/2D topology, PoA dropped 2.2x and TTFT P99 dropped 7.6x. These results demonstrate that game-theoretic insights can yield practical performance gains for large-scale LLM serving.