RATrain boosts LLM training on bandwidth-constrained supercomputers 1.35x

Existing GPU-oriented training runtimes struggle on bandwidth-constrained heterogeneous supercomputing platforms like the MT-3000, which lacks high-bandwidth device memory, fast interconnects, and mature collective communication libraries. To address this, the team developed RATrain, which formulates standard non-interleaved 1F1B training as a training-state lifecycle scheduling problem. It orchestrates gradient synchronization, parameter updates, parameter-view prefetching, and activation recovery at layer-level and stage-local granularity. A custom MT-3000-aware execution backend handles FP16 GEMM, Attention Backward, and explicit data movement, while a resource-aware planner selects feasible configurations under the strict 20GB DDR limit per compute cluster.

RATrain was evaluated on a real MT-3000 platform using LLaMA-2-7B, Baichuan2-13B, Qwen2.5-32B, and LLaMA-2-70B. Results show up to 1.35× end-to-end speedup over MT-3000-adapted GPU-style strategies. For LLaMA-2-7B, RATrain scales to 1,024 clusters, reaching 112,790.55 tokens/s with 97.0% scaling efficiency. A correctness run over 1.028 billion tokens confirmed that RATrain preserves the loss trajectory of a baseline run, with a maximum relative loss deviation of only 0.081%. This demonstrates that resource-aware scheduling can unlock efficient LLM training on previously incompatible supercomputing hardware.