New AI Training Trick Is 4x Faster and Loses Nothing

Training a big AI model is like coordinating a massive team, except the team is thousands of computer chips that need to share information constantly. When a model uses a Mixture-of-Experts design—a common way to make AI smarter by dividing work among smaller experts—the communication between chips can take more time than the actual computations. Engineers call this overhead the Allgather step, and it has become a major bottleneck.

A team at the University of Virginia, in collaboration with NVIDIA researcher Masahiro Tanaka, noticed something interesting: after a short warm-up, the parts of the numbers that define how big or small they are (called exponents) almost never change from one training step to the next. So, instead of sending those exponents every time, they kept a copy on each chip and only transmitted the remaining details. Those details, when combined with the cached exponents, perfectly recreate the original numbers. The process is lossless, meaning no accuracy is lost—every value is exactly what it would have been without this shortcut.

The savings are large. In real tests training the OLMoE-1B-7B model on 16 GPUs, CIERA ran 3.7 times faster than both the original method and an older lossy approximation. Scaling to 128 GPUs, the researchers project a 4.4x speedup while still guaranteeing bitwise-exact results. They also made sure to compress only when it actually saves time, since some layers aren't worth the effort. The outcome is a simple, drop-in technique that reduces cost, time, and energy needed to build advanced AI.