NEURON-Fabric: CXL-side low-bit aggregation slashes gradient traffic to 5%
CXL memory controller aggregates gradients in 5 cycles, cutting communication by 95%.
NEURON-Fabric, a new architecture from Wang, Huang, and Lung, leverages the Compute Express Link (CXL) memory controller to perform low-bit gradient aggregation during distributed training of large models. By intercepting gradient cache lines as they pass through CXL-attached memory, the controller can apply gradient-binary (G-Binary) sign-count aggregation or gradient-ternary (G-Ternary) gated aggregation, drastically reducing the bandwidth needed for All-Reduce communication. Cycle-level simulations show the low-bit datapath adds at most 1.67% exposed runtime overhead under worst-case cache misses, and is often hidden by CXL service latency.
The system includes a control interface to selectively apply low-bit approximation per layer or phase. Experiments on CIFAR-10/ResNet-18 and SST-2/DistilBERT show low-bit aggregation closely matches FP32 accuracy, while full-path low-bit fails on CIFAR-100/ResNet-18. A layer-aware admission policy identifies the classifier head as sensitive, preserving FP32 there and applying low-bit to the backbone, recovering accuracy while cutting gradient traffic to just 3.6–5.4% of the FP32 baseline. Hardware synthesis estimates show the 512-bit aggregation datapath is small enough to be a near-memory extension, not a separate accelerator.
- Gradient traffic reduced to 3.6–5.4% of FP32 baseline via low-bit aggregation (G-Binary and G-Ternary).
- Runtime overhead only 1.67% in worst-case cache miss regime; often hidden by CXL service time.
- Layer-aware admission keeps classifier head on FP32, maintaining accuracy while applying low-bit to backbone.
Why It Matters
NEURON-Fabric could slash distributed training communication costs, enabling faster scaling of LLMs with minimal hardware changes.