PyTorch fixes critical NCCL collective split bug
A PyTorch patch prevents tensor splits from breaking NCCL collective operations with negative or malformed inputs.
PyTorch just fixed an all-to-all split-size validation issue across its distributed backends. The nccl2 path was casting split sizes directly to unsigned values and only normalizing when both split lists were empty, so negative values wrapped and malformed lists reached NCCL. Gloo and stock NCCL also skipped leading-dimension validation when both lists were empty, allowing equal splits that didn’t divide the tensor along dimension zero. The fix applies shared c10d split checks to every Gloo, stock NCCL, and nccl2 configuration before launch, and independently normalizes empty nccl2 input and output split lists — giving all three implementations the same validation and equal-split semantics.
- Bug affected all-to-all collective operations in PyTorch’s NCCL backend, risking tensor misalignment in distributed training
- Patch standardizes split-size validation across Gloo, stock NCCL, and nccl2 implementations
- Users should update to PyTorch nightly builds; test plan includes NCCL_DEBUG=WARN and CUDA_VISIBLE_DEVICES checks
Why It Matters
Prevents silent data corruption in multi-GPU training runs by ensuring tensor splits align correctly across distributed systems.