Mixed-precision CA-SGD speeds up GPU training 6x with minimal accuracy loss
Researchers cut communication overhead in distributed SGD by 5–6x on A100 GPUs.
Researchers from NERSC and Cornell have published a new mixed-precision variant of communication-avoiding stochastic gradient descent (CA-SGD) that dramatically reduces the communication bottleneck in distributed training of generalized linear models on GPU clusters. Their method, described in a recent arXiv preprint, replaces s consecutive AllReduce synchronization steps with a single AllReduce of an sb×sb Gram matrix. This amortizes communication overhead, but historically increased computation and bandwidth – a trade-off now mitigated by modern GPU matrix hardware and reduced-precision formats like BF16.
The team’s finite-precision analysis decomposes the local rounding error into nine independent precision choices, allowing the recipe to transfer across GPU generations. On NERSC Perlmutter A100 GPUs, mixed-precision CA-SGD matched FP32 SGD loss within 0.5% on logistic, linear, and Poisson problems. Performance gains reached 5.1–6.8× speedup over standard FP32 SGD on datasets including epsilon, SUSY, HIGGS, and synthetic Poisson benchmarks. The software is open-sourced, and the approach is particularly impactful for large-scale regression and classification tasks where communication dominates training time.
- Replaces s AllReduces with one Gram matrix AllReduce, cutting synchronization points by up to 6× per outer iteration.
- Uses BF16 for input storage and Gram GEMM, while communicating in high precision and accumulating in high precision.
- Achieves 5.1–6.8× speedup over FP32 SGD on A100s with less than 0.5% loss in accuracy across multiple GLM tasks.
Why It Matters
Faster distributed training of linear models reduces cloud GPU costs and enables larger-scale ML experiments.