New CUDA kernels speed up LLM inference 2.47x by fusing SwiGLU activation

A team of researchers (Jangda et al.) has published a paper demonstrating two novel CUTLASS-based kernels for efficient LLM inference. The key innovation addresses the SwiGLU activation function, which is dominant in modern LLMs like Qwen-2.5. SwiGLU requires materializing an intermediate tensor that costs 9-37% of total MLP execution time. The researchers designed two complementary kernels: Kernel-1 overlaps the Swish computation on the Gate accumulator with Up-tile loading using a Pingpong warp-specialized schedule; Kernel-2 interleaves SwiGLU with tile stores via a custom Epilogue Visitor Tree. By fusing activation into the matrix multiply at the tile level, these kernels shift the workload from memory-bound to compute-bound, dramatically improving utilization.

The results are impressive: on Qwen-2.5 models ranging from 0.5B to 72B parameters running on NVIDIA H100 GPUs, the fused kernels achieve up to 2.47x speedup over standard PyTorch implementations. They reach 79.5% peak BF16 utilization. Notably, the researchers found that NVIDIA's cuBLAS library cannot replicate this tile-level fusion – cuBLAS implementations are 3-7x slower and introduce 4.5-11% numerical mismatches, whereas the new kernels produce zero mismatches. This validation underscores the need for hand-crafted, tile-level design to fully exploit hardware for LLM inference optimization. The work has been submitted to arXiv under the Distributed, Parallel, and Cluster Computing category (arXiv:2607.02521).