PyTorch PR #192530 fixes rank 0 multicast bottleneck with fabric handles
Rank 0's memcpy serialized behind host transfers—now it uses peer copy path.
In PyTorch's symmetric memory (symm_mem) layer, the initialization of multicast for a block had a subtle asymmetry: rank 0 retained the CUmemGenericAllocationHandle directly from cuMulticastCreate, while every other rank adopted an imported handle via fabric export/import. That provenance matters because the driver selects a memory copy path based on how the handle was created. With a natively created handle, writes into the multicast virtual address are treated as plain local device-to-device copies, scheduled on the same async copy engines that service host-to-device and device-to-host transfers. As a result, the memcpy in symm_mem::memcpy_to_multicast_ would serialize behind any concurrent host transfer on rank 0.
By having rank 0 self-import its own exported handle, the driver now uses the peer copy path (Memcpy PtoP) on a separate engine, allowing overlap with host traffic. The fix was validated on a 2-GPU GB200 with fabric/IMEX enabled, where the multicast handles are exchanged as CU_MEM_HANDLE_TYPE_FABRIC. The profiler shows rank 0 flipping from 'Memcpy DtoD' to 'Memcpy PtoP' across 5 runs of _low_contention_all_gather_ce_multicast_out, matching rank 1's behavior exactly. This removes an unexpected performance cliff in multi-GPU collectives, especially in fabric-attached clusters like GB200.
- Rank 0 in PyTorch's symm_mem used a natively created CUDA multicast handle, causing memcpy to serialize behind host transfers on the async copy engines.
- The PR makes rank 0 self-import its exported fabric handle, switching to the peer copy path (Memcpy PtoP) on a separate engine.
- Tested on 2-GPU GB200 with fabric/IMEX: rank 0's memcpy entries now match rank 1 exactly, all using 'Memcpy PtoP'.
Why It Matters
Removes a hidden multi-GPU performance bottleneck, enabling faster and more scalable distributed training on fabric-linked systems.