PyTorch Monarch on AMD ROCm enables fault-tolerant training at scale
Single-controller distributed training now resilient to node failures on AMD GPUs.
Deep Dive
PyTorch Monarch has been ported to AMD Instinct GPUs with ROCm, enabling elastic, fault-tolerant distributed training. The system recovers from node failures via a hierarchical supervision tree, with local restarts taking seconds and minimizing wasted computation during large-scale LLM training.
Key Points
- Achieves 96.16% scaling efficiency on 1,024 AMD MI325 GPUs for DeepSeekV3-671B training.
- Recovers from node failures in seconds via hierarchical supervision tree, without restarting the entire job.
- Ported to ROCm by converting CUDA to HIP with hipify_torch, using RCCL and preserving RDMA support.
Why It Matters
Enables reliable large-scale AI training on AMD hardware, reducing downtime and wasted compute.