Developer Tools

PyTorch PR adapts FlexGEMM grouped epilogues to NVGEMM for faster GPU inference

PyTorch's new PR fuses addmm/baddbmm epilogues into NVGEMM, cutting GPU kernel overhead

Deep Dive

PyTorch's FlexGEMM infrastructure just took a step forward with PR #190824, which adapts grouped epilogues to work with NVGEMM, NVIDIA's optimized GEMM path. Authored by mlazos with an AI assistant, the change decomposes fused operations like `addmm` and `baddbmm` into simpler `mm` or `bmm` calls followed by explicit alpha, beta, and bias operations. That decomposition lets Inductor's scheduler analyze and fuse those epilogues as ordinary NVGEMM epilogues, unlocking better CPU-side scheduling and potentially faster GPU kernels.

The PR also carefully preserves beta-zero semantics — an important edge case where the bias term is mathematically irrelevant — by omitting the bias contribution entirely. This avoids unnecessary memory reads and arithmetic. For kernel shapes that aren't yet supported by the NVGEMM path, the change falls back gracefully to existing selection logic, ensuring correctness across a wide range of GEMM configurations. The PR is part of a larger ghstack dependency chain (including #190809, #190810, #190817, and others), signaling an ongoing effort to modernize PyTorch's GEMM code generation for NVIDIA hardware. For practitioners running transformer inference or training loops, this means more efficient fused matrix multiplications under `torch.compile`, with less overhead and better kernel fusion potential.

Key Points
  • Decomposes `addmm` and `baddbmm` into `mm`/`bmm` plus explicit alpha, beta, and bias ops for NVGEMM fusion
  • Preserves beta-zero behavior by skipping bias contribution entirely, saving memory and compute
  • Unsupported kernel shapes route to existing NVGEMM selection and fallback paths, maintaining correctness

Why It Matters

GEMM-heavy PyTorch workloads on NVIDIA GPUs get better kernel fusion, reducing overhead and speeding up training and inference.

📬 Get the top 10 AI stories daily