Zellige scheduler boosts image-video AI training by 1.54x

Training diffusion transformers (DiTs) on mixed image and video data is essential for high-quality video generation, but it poses a challenging mixed-length sequence problem across GPUs. Existing systems use data parallelism (DP), context parallelism (CP), or a combination, which the authors model as disjoint-group placement. They prove these designs face a fundamental tradeoff: inter-group load imbalance versus intra-group communication redundancy. Zellige, from Guangyu Xiang and colleagues, solves this with moldable sequence placement, jointly selecting each sequence's parallelism configuration and participating ranks.

Zellige has three components: a hardware profiler that estimates execution time and memory for candidate placements, a two-stage planner that balances compute-heavy anchor sequences and packs lighter filler sequences into residual capacity, and a coalesced attention engine that efficiently runs whole sequences alongside distributed-attention shards. The profiler is remarkably accurate, predicting step makespan and peak memory with mean absolute percentage errors of 3.4% and 1.5% across 21 plans. The planner solves each batch in 33-119ms, much faster than a joint-placement reference, while modeled makespans differ by at most 0.32%.

In end-to-end evaluations, Zellige outperforms KnapFormer by 1.12-1.48x on 16 A800 GPUs and 1.27-1.54x on 32 A6000 GPUs, demonstrating up to 35% faster training. This is a significant practical advance for scaling video generation models, where training efficiency directly impacts cost and iteration speed.