X-Stage pipeline boosts DiT inference up to 1.62x on NVIDIA GPUs

Researchers from an academic team have introduced X-Stage, a previously overlooked pipeline stage that enables fine-grained control of communication-computation overlap in distributed diffusion transformer (DiT) inference. Modern GPU kernels can issue remote stores directly, but existing systems only schedule when communication begins and when data is consumable, ignoring the critical post-issue progress before remote completion. This omission makes sender backpressure unpredictable and limits performance. The team's measurements on an eight-GPU node with NVIDIA's latest architecture revealed two behaviors: short remote-store bursts drain quickly as the issuer resumes work, while sustained injection exhausts finite outstanding capacity and delays subsequent issues.

To model this, they developed a lightweight Burst-Gap model parameterized by backpressure-free issue time, effective drain rate, and outstanding capacity. Guided by this model, they redesigned two communication-computation fused kernels. For DeepGEMM MegaMoE, interleaving Linear-1 and Linear-2 work across expert waves places computation between concentrated remote-store bursts, yielding a 1.18x geometric-mean and 1.62x maximum kernel speedup over the Expert-Wave baseline across 84 configurations. For Ulysses sequence-parallel attention, tile-granular fusion of the post-attention All-to-All with FlashAttention lets an output-tile owner issue remote stores and resume computation without a dedicated communication warp or streaming multiprocessor. FlashAttention-3 and FlashAttention-4 reached maximum sender-visible speedups of 1.43x and 1.42x over serial execution, and at long sequences their steady-state times approached those of FlashAttention alone. These results establish post-issue progress as a measurable scheduling lever for shaping bursts, avoiding backpressure, and hiding sender-side overhead.