PyTorch adds 128x256x64 matmul config boosting H100 performance up to 1.78x

PyTorch’s latest commit to its Inductor compiler adds a new CUDA matrix multiplication configuration specifically optimized for Hopper architecture GPUs (e.g., H100). The new config, defined as GemmConfig(128, 256, 64, 4, 8), uses a tile size of 128x256x64 with NS=4 (number of splits along the reduction dimension). This configuration is designed to better exploit the larger shared memory and compute capabilities of Hopper-class GPUs compared to the existing set of 20 configs, which tended to under-explore larger M-tile shapes. The change was merged into the CUDAConfigHeuristic.mm_configs list, and autotune per shape selects the best config, so smaller or irregular shapes still use existing alternatives.

The performance impact is significant for large-scale matrix multiplications common in training transformers and other deep learning models. Across six production shapes with N ranging from 256 to 16384 and K from 512 to 3072, the new config wins where autotune picks it, showing speedups between 1.16x and 1.78x. On a target shape of M=8192, N=6144, K=3072 (fp16), a controlled test on an isolated H100 SXM ran 15 trials — all 15 favored the new config with a median 1.343x speedup. An M-sweep across sizes from 1024 to 32768 showed stable 1.30–1.35x gains for M ≥ 2048, while M=1024 correctly fell back to the original config (no regression). Compile time increases by roughly 5% due to one extra candidate per autotune session, a negligible tradeoff for the runtime gains.