CTA-Pipelining slashes multi-GPU LLM latency by 31%
New technique cuts MLP inference latency by up to 31.8% on 8-GPU H200 systems
As multi-GPU systems evolve into tightly integrated shared-memory structures, software still treats coherent interconnects as high-speed networks. The shift to latency-constrained LLM serving has exposed the limits of Tensor Parallelism (TP). Researchers from UIUC and NVIDIA introduce CTA-Pipelining, a novel execution paradigm that leverages Cooperative Thread Array (CTA) dependencies to run dependent kernels concurrently across GPUs. This spatial scaling method directly addresses the latency bottleneck in multi-GPU inference.
Tested on 8-GPU NVIDIA H200 and B200 systems, CTA-Pipelining reduced latency for 2-layer GEMM—the core of MLP layers—by up to 31.8% compared to micro-batching, and 29.6% compared to standard TP. Crucially, it works as an orthogonal dimension to TP, enabling hybrid approaches that push latency boundaries further. The method is implemented using CUTLASS, cuBLAS, and NCCL libraries, making it practical for existing deployment stacks.
- Up to 31.8% latency reduction on 2-layer GEMM (MLP) vs micro-batching on 8-GPU H200/B200 systems
- 29.6% reduction compared to standard Tensor Parallelism, with potential for further gains when combined
- Utilizes CTA-level dependencies to overlap kernel execution across GPUs in shared-memory multi-GPU setups
Why It Matters
Cuts LLM serving latency by ~30% without new hardware—critical for real-time AI applications like chatbots and agents.