MegaSlide-DiT adapts 105B video models on a single H200 GPU
No more GPU clusters: run massive video diffusion models on one workstation with 1.5TB RAM.
MegaSlide-DiT tackles the memory wall that plagues large video diffusion Transformers. A typical 100B+ DiT needs over a terabyte of persistent state, and naive spatiotemporal self-attention grows quadratically with sequence length. To overcome this, the system keeps all persistent weights, master weights, and optimizer moments in host RAM, streaming only transient shards to the GPU on demand. This reduces GPU memory requirements dramatically while still enabling full-parameter adaptation. The second innovation, 3D Deformable Slide Attention (3D-DSA), replaces quadratic global attention with a motion-adaptive local operator that scales linearly with sequence length, further reducing both memory and computation.
The authors demonstrate that a pre-trained 105B parameter DiT can be adapted on a single H200 with 1.5TB of host RAM, a feat previously requiring multiple GPU nodes. While it doesn't magically eliminate bandwidth limits or train from scratch, MegaSlide-DiT offers a pragmatic path for researchers to fine-tune state-of-the-art video generation models on high-end workstations. Detailed memory accounting and execution traces validate the design, making this a significant step toward democratizing massive generative model adaptation.
- Streams model parameters from 1.5TB host RAM to GPU on demand, avoiding GPU memory overflow
- 3D Deformable Slide Attention reduces memory and compute complexity from quadratic to linear in sequence length
- Adapts a 105B-parameter video DiT on a single H200 GPU, previously requiring multi-GPU clusters
Why It Matters
Democratizes large video diffusion model adaptation without requiring massive GPU clusters.