Amazon and NVIDIA Fixed the Glitch That Wastes Weeks of AI Training

NVIDIA's Resiliency Extension (NVRx) can be integrated into PyTorch Fully Sharded Data Parallel training on Amazon EKS to keep large distributed training jobs making progress when things break. At the scale described — jobs running for hours or days across dozens of nodes — interruptions are statistically inevitable, and a single GPU fault can cascade through NCCL timeouts, crash pods out of sync, and burn expensive GPU hours while making zero training progress. NVRx adds fault-tolerance primitives as ordinary pip-installable Python imports, with no custom kernels, no PyTorch fork, and no recompile: async checkpointing that overlaps I/O with training, in-process restart that recovers from faults in seconds without touching the container lifecycle, and ft_launcher for automatic worker respawn on hard crashes like SIGKILL and OOM kills. On the cluster sizes covered in the article, synchronous checkpointing alone consumed up to 40% of total wall time.