New gradient-based checkpointing cuts AI training time by up to 89%

Distributed training of large deep-learning models is increasingly common, but system failures force expensive checkpointing to enable recovery. Frequent checkpointing reduces recovery time yet creates massive storage and I/O overhead, slowing training. A team led by Chenxuan Yao (Huazhong University of Science and Technology) found a lever: gradients produced during backpropagation can themselves serve as differential checkpoints—and are smaller than the parameter deltas typically used. Their framework, referred to as "sysname," reuses compressed gradients in compression-enabled training systems, removing the need for separate differential computation and cutting checkpoint transmission costs.

The authors extend this idea to scenarios without gradient compression in "sysnameplus," which combines layer-wise-reuse snapshotting and incremental-merging persistence to overlap checkpointing with training execution. Evaluations across diverse workloads, including billion-parameter-scale models, show both variants support checkpointing at frequencies as high as once per iteration. Training time drops by up to 89.2% with the gradient-reuse approach and 81.2% with the layer-wise variant, compared to baseline frequent-checkpointing methods. This work, available as arXiv:2509.04084, offers a practical path to near-zero-overhead checkpointing for large-scale distributed training.