PyTorch's new XPU fusion speeds up Llama inference by 7-15%

PyTorch has merged a commit (ciflow/xpu/181854) that auto-enables the batch_linear_lhs pre-grad fusion pass when compiling models for XPU (Intel GPU) devices. The optimization targets parallel linear layers sharing the same input—such as the gate_proj and up_proj in Llama's MLP block. Instead of launching two separate kernel calls, the compiler fuses them into a single wide GEMM operation, then splits the result. This yields three key benefits: (1) reducing kernel launch overhead from two launches to one (saving ~5μs of dispatch cost), (2) leveraging a larger N dimension for better GPU occupancy and wave efficiency, and (3) reading the input tensor only once instead of twice. Benchmarks show inference speedups of 7–15% on XPU hardware.

The implementation carefully uses a local copy of fusion_options to avoid mutating the global configuration, preventing interference with tests that explicitly set pre_grad_fusion_options={}. This commit, tagged on June 7, was co-authored with Copilot and builds on existing PyTorch inductor infrastructure. For developers deploying LLMs on Intel GPUs, this means faster token generation without any code changes—just upgrade to the latest PyTorch and recompile. The optimization is particularly impactful for production inference where every millisecond counts, and it demonstrates PyTorch's continued investment in hardware-specific compiler passes for XPU.