PyTorch merges NHWC GroupNorm CUDA kernel for faster convnets

PyTorch has merged a new CUDA implementation of GroupNorm that supports the NHWC (channels-last) memory format. Previously, GroupNorm kernels forced outputs into a channels-first (NCHW) contiguous layout, which is suboptimal for convolutional neural networks that commonly operate in channels-last format for better cache locality and speed. The change, described in PR #191945, reuses existing GroupNorm infrastructure rather than building a separate solver, and the author notes it achieves comparable memory bandwidth on NVIDIA H100 GPUs. The real win is eliminating the costly data layout conversion and the resulting performance penalty for convnet pipelines.

The commit, tagged on trunk as 8003932, was authored with assistance from Codex and approved by ngimel, a key PyTorch CUDA maintainer. While the implementation is not a major bandwidth breakthrough—it matches existing memory throughput—it removes a structural inefficiency that forced channels-last tensors to be copied or transformed just to run GroupNorm. For practitioners using convnets with NHWC tensors, this means fewer memory stalls, reduced overhead, and smoother integration of normalization layers. The change is especially relevant for high-throughput training and inference on H100-class hardware, where memory layout conversions can become a bottleneck. As PyTorch continues to optimize for modern GPU architectures, this patch is a small but meaningful step toward making channels-last the default path for vision models.