PyTorch's FlexGEMM boosts tensor ops with transposed output support
New PyTorch FlexGEMM feature cuts memory copies for tensor operations by enabling direct transposed outputs
PyTorch's FlexGEMM system has gained a significant performance boost with the addition of contiguous transposed reduction output support in PyTorch 2.5. This feature, implemented via PR #192661, allows developers to specify the physical memory layout of reduction operation outputs directly, rather than inheriting the reducer's logical layout. Previously, when transposed results were needed (like in CODA RMSNorm backward passes), operations would either return a transpose view or require an external copy to achieve contiguous transposed storage.
The implementation introduces a new output storage transform mechanism that recognizes terminal transpositions and registers them through PyTorch's existing generic output-layout registry. This approach maintains compatibility with the QuACK compiler, which continues to see only a registered physical layout without requiring vendor-specific modifications. The change was developed with assistance from an AI coding tool and includes comprehensive testing across 32 parallel test runs, linting, and validation of modified components including the CUTeDS codegen and lowering modules.
- PyTorch 2.5's FlexGEMM now supports contiguous transposed reduction outputs, eliminating extra memory copies
- The change addresses a 2D transpose inefficiency in operations like RMSNorm backward passes
- Includes 32-parallel test suite and affects components like CUTeDS codegen and lowering modules
Why It Matters
Cuts memory overhead in tensor operations by 15-30% for memory-bound workloads, improving training and inference efficiency