New FP4 method enables stable LLM training with <1.3% accuracy loss

A team of researchers (including Mehdi Rahimifar, Amin Darabi, and others) have introduced a new training framework that makes 4-bit floating point (FP4) training stable for large language models. The key innovation is a 2D block quantization scheme that enforces consistency between forward and backward passes—addressing a fundamental flaw in prior 1D approaches where transposed tensors received different scaling factors, biasing gradients. By combining this with truncation-free scaling and stochastic rounding, the method maintains unbiased gradients and controls quantization error. For attention mechanisms, which are especially sensitive to precision, they use MXFP8 quantization for query/key projections, resulting in a practical mixed-precision design.

The method was validated on dense LLMs up to 7B parameters and a 30B Mixture-of-Experts model trained on up to 100B tokens. Across all settings, it achieved stable end-to-end FP4 training and closely matched BF16 performance, with less than 1.3% degradation in perplexity and downstream accuracy. This represents a significant step toward making LLM training more efficient by halving memory and compute requirements compared to FP8, without sacrificing model quality. The work is published on arXiv (2607.24953) and offers a simple, effective pathway for scaling low-precision training.