AoiZora cuts video diffusion inference latency 1.42x on TPU v5e

Video diffusion models are increasingly critical for generative AI serving, but their heavy denoising iterations over large spatio-temporal latents make single-device inference impractical. Distributing inference across multiple accelerators is essential, and TPU sub-slices have emerged as a popular choice. However, existing auto-parallel systems search over logical device meshes without accounting for the physical layout of the TPU interconnect, leaving significant performance on the table.

Enter AoiZora, a topology-aware compiler planner from Rice University and collaborators. It first eliminates weak sharding candidates using inexpensive pre-compilation IRs, then compiles only the surviving plans and ranks them using compiled HLO plus a topology-aware communication model. On TPU v5e sub-slices running the Wan 2.1 model, AoiZora achieves up to a 1.42x speedup on one-step denoising latency—all without altering model code, compiler lowering, or collective kernels. This work bridges the gap between logical sharding and physical placement for low-latency video diffusion inference.