MegaTrain: Full Precision Training of 100B+ Parameter Large Language Models on a Single GPU

A team of researchers led by Zhengqing Yuan has introduced MegaTrain, a groundbreaking system that fundamentally rethinks how large language models are trained. Instead of the traditional GPU-centric approach that struggles with memory constraints, MegaTrain adopts a memory-centric architecture where model parameters and optimizer states reside in host (CPU) memory. The GPU is treated as a transient compute engine that streams in parameters layer by layer, computes gradients, and streams them back out. This dramatically reduces the persistent state that must be stored on the GPU, enabling the training of massive 100B+ parameter models like GPT-4 on hardware that was previously insufficient.

To overcome the inherent CPU-GPU bandwidth bottleneck, the team implemented two key optimizations. First, they developed a pipelined double-buffered execution engine that overlaps parameter prefetching, forward/backward computation, and gradient offloading across multiple CUDA streams, keeping the GPU continuously busy. Second, they replaced PyTorch's persistent autograd graphs with stateless layer templates, dynamically binding weights as they stream in. This eliminates the memory overhead of graph metadata and provides greater scheduling flexibility. On a single NVIDIA H200 GPU paired with 1.5TB of host memory, MegaTrain reliably trains models up to 120 billion parameters. In benchmarks, it achieved 1.84 times the training throughput of Microsoft's DeepSpeed ZeRO-3 with CPU offloading when training a 14B parameter model, and it also enables training a 7B model with an extremely long 512k token context on a single GH200 superchip.