GIFT cuts LLM pretraining time by 7.6% with geometry-aware gradient compression
New method makes FP8 gradient communication as accurate as full-precision for LLMs.
Gradient communication is a major bottleneck in scaling LLM pretraining. Low-precision formats like FP8 and NVFP4 can dramatically reduce communication volume, but existing quantization methods in Euclidean space degrade model performance because highly anisotropic gradients suffer direction-dependent distortion. GIFT (Geometry-Informed Low-precision Gradient Communication) addresses this by first transforming gradients into a nearly isotropic coordinate system before quantization. This makes low-precision representations much more faithful to their high-precision counterparts. The method only changes the coordinate system for communication—it does not alter the optimizer, training recipe, communication collective, or the low-precision format itself.
To keep overhead manageable, GIFT incorporates a simplified geometry-aware transformation using low-rank approximation and selective application. Tested on Llama-300M and Llama-600M models across 64 NVIDIA GH200 Superchips, GIFT achieved a 7.6% reduction in end-to-end pretraining time for the larger model while improving downstream task accuracy retention compared to direct FP8 communication. The paper includes 12 pages of analysis with convergence guarantees. This approach is particularly promising for distributed training clusters where gradient communication dominates wall-clock time, offering a plug-in improvement that requires no changes to existing infrastructure or training pipelines.
- GIFT transforms gradients into near-isotropic space before quantization to reduce direction-dependent distortion from anisotropic gradients.
- Achieves 7.6% faster end-to-end pretraining for Llama-600M on 64 NVIDIA GH200 Superchips vs. direct Euclidean FP8 communication.
- Uses low-rank approximation and selective application to balance computation overhead with communication reduction; leaves optimizer and training recipe unchanged.
Why It Matters
A plug-in method that speeds large-scale LLM pretraining by up to 7.6% without modifying existing infrastructure or training recipes.