Meta's FlashNormAttention fuses normalization into GEMM, achieving up to 90% latency hiding
New kernel fusion hides 90% of normalization latency by fusing with GEMMs on B200 GPUs.
Meta's new kernel fusion techniques, presented in 'Towards Free Normalization,' tackle the memory-bound bottleneck of normalization ops (LayerNorm, RMSNorm) in LLMs and recommendation models. The challenge arises from differing tiling strategies between normalization (reduction across a dimension) and matmul operations. The authors propose Lazy Pre-Norm and Multi-CTA Norm Fusion, fusing norms with GEMMs to hide up to 90% of normalization latency. This enables better hardware utilization, as normalization can take 10-20% of total training time.
FlashNormAttention goes further by fusing both LayerNorm and RMSNorm into attention kernels like GDPA, achieving up to 35% kernel speedup on NVIDIA B200 GPUs. The techniques are implemented with two DSLs: TLX (low-level Triton extensions for hardware-aware control) and Helion (high-level DSL for portability and autotuning). This work directly improves throughput for Meta's largest models, including the Generative Ads Model (GEM), by reducing memory-I/O overhead without compromising accuracy.
- Meta's kernel fusion techniques (Lazy Pre-Norm, Multi-CTA Norm Fusion) hide up to 90% of normalization latency when fused with GEMMs.
- FlashNormAttention fuses LayerNorm and RMSNorm into attention kernels, achieving up to 35% speedup on NVIDIA B200 GPUs.
- Implemented using TLX (Triton-based low-level DSL) and Helion (high-level portable DSL) for broad applicability.
Why It Matters
Boosts training throughput for LLMs and recommendation models by eliminating memory-bound normalization bottlenecks.