AMD GPUs now train AI models 13.4% faster with FP8

At the PyTorch Conference 2025, AMD demonstrated FP8 training performance improvements that have now been upstreamed into TorchAO and TorchTitan, enabling native support for AMD Instinct GPUs. The optimizations deliver a 13.4% throughput gain over BF16 on dense models like Llama3-8B while maintaining competitive memory usage (~39GB). For Mixture-of-Experts architectures such as DeepSeek-V3 671B, the team developed fused Triton quantization kernels that recovered 89% of FP8 overhead, with individual kernel optimizations achieving up to 6.2× speedups.

The breakthrough required three key components: native AMD FP8 format support via the FNUZ variant (Finite, No NaN, Unsigned Zero), grouped GEMM enablement for MoE models on ROCm, and a Triton fusion pipeline to reduce quantization overhead. Critical correctness fixes were implemented to address format mismatches where TorchAO initially used NVIDIA's e4m3fn format against AMD's FNUZ max value of 240, which could silently corrupt gradients. The team added hardware auto-detection to ensure proper format selection across AMD's MI300X, MI325X, and MI350X GPUs.