SAGE boosts spiking transformer accuracy by 1-2% via attention-guided gradients

SAGE (Surrogate-gradient Adaptation via Attention-Guided Entropy) is a new training mechanism for spiking Transformer networks developed by Kiran Nair, Rodrigue Rizk, and KC Santosh. SNNs are energy-efficient alternatives to conventional deep networks because they use sparse, event-driven computation, but training them is notoriously difficult: the spike function is non-differentiable, so surrogate gradients are used—and their fixed shape can be suboptimal across layers and training stages. SAGE solves this by estimating block-level uncertainty from normalized self-attention entropy and using that signal to dynamically adjust the surrogate gradient slope during training.

Crucially, the method only modulates the training-time surrogate parameter, so the original architecture and deployment cost remain unchanged. In experiments on CIFAR-10 and CIFAR-100, SAGE delivered consistent accuracy improvements of 1-2% over fixed-surrogate baselines across several simulation time steps. The paper, currently in review at a conference, demonstrates that attention-derived uncertainty is a lightweight and effective training signal for adaptive surrogate-gradient learning in transformer-based SNNs.