PyTorch's Inductor Compiler Fuses Kernels for 10x GPU Speedup

PyTorch's torch.compile feature leverages kernel fusion to dramatically accelerate GPU workloads. The Inductor compiler automatically identifies chains of pointwise operations—common in neural network layers—and merges them into a single Triton kernel. Without compilation, each torch operation launches a separate kernel, incurring overhead and forcing intermediate results to be written to and read from slow global memory. For example, a sequence of multiply, add, and sigmoid normally spawns three kernels with eight memory operations.

With fusion, a single kernel loads all inputs once (five loads), performs all arithmetic in fast registers, and writes only the final output. This eliminates two intermediate writes and reduces global memory traffic by 37.5%. The result: 66% fewer kernel launches and significantly less memory bandwidth pressure. For practitioners, this means up to 10x speedups on typical model code without any manual optimization—just add torch.compile(). The technique, called vertical fusion, is especially powerful for deep learning where operations naturally form dependency chains.