FP8 attention precision saved: S=256 scale prevents collapse, says new paper

A new arXiv paper by Reed Lau (submitted June 2026) dives into the precision challenges of using FP8 (E4M3) for attention computation, specifically focusing on the casting of the softmax probability matrix (P) before the P×V matrix multiplication. The study zeroes in on two implementation choices that affect output precision: the KV block iteration order and the static scaling factor applied to P before casting. The paper reveals that forward iteration through KV blocks leads to 'P-collapse'—a phenomenon where a fraction of non-sink P values underflow to zero, mathematically characterized by a threshold Δ_c = 6.93 + ln S – δ_k. Reverse iteration completely removes this underflow when combined with S = 256.

The paper further provides a constructive proof that S = 256 (2⁸) is the optimal static scale for FP8 E4M3: it is bit-exact in IEEE 754 scaling, lies on the lower envelope of the sawtooth quantization error function (with minimum worst-case step dp = 2⁻⁴), and achieves maximum normal-range coverage among all bit-exact (2^k) scales. Although a non-bit-exact scale like 448 offers slightly higher coverage, the authors argue that S=256 is the best practical choice. These optimizations are already deployed in FlashAttention-3/4 based on engineering heuristics; this paper provides the first rigorous quantitative account. Kernel-faithful experiments with Q, K, V kept in FP32 to isolate the P-cast effect show 3–10x MSE improvement at moderate sink strengths, with both fixes saturating to the same precision floor when combined.