NVIDIA NCCL achieves near speed-of-light GPU latency with new collectives

GPU collective communication has long been optimized for bandwidth, but emerging workloads—particularly long-context, decode-heavy LLM inference—are increasingly latency-bound. In serving large models across multiple GPUs, many small collectives lie on the critical path of token generation, meaning even a microsecond of overhead directly impacts performance and cost. This new paper from a team including NVIDIA researchers systematically studies how to approach the hardware speed-of-light (SoL) lower bound for GPUs within a scale-up network.

The team identifies three key principles for near-optimal designs: barrier-free synchronization, efficient use of symmetric memory, and multicast. Building on NCCL's device-side API, they develop low-latency interfaces for constructing custom collective kernels and implement new symmetric collectives. Microbenchmarks show latency reductions for small and medium messages, bringing overhead to within 7% of the absolute SoL lower bound—a dramatic improvement over existing implementations.

When integrated into real applications, these kernels improve inter-token latency and throughput in LLM inference and accelerate cuSOLVERMp (an HPC solver library). The work demonstrates that minimizing microsecond-scale latency can yield tangible performance gains for both AI inference and traditional high-performance computing, making it highly relevant for anyone deploying large-scale GPU clusters.