SMM Transformer matches ANN accuracy with 97% less attention energy

Deep learning's relentless accuracy gains often come at the cost of energy-inefficient dense computation. A new arXiv paper from researchers including Xiubo Liang and Hongzhi Wang proposes SMM Transformer, a spiking neural network (SNN)-based multimodal Transformer framework designed to make attention mechanisms drastically more energy-efficient without sacrificing accuracy. SNNs promise event-driven, sparse computation, but building deep multimodal Transformers on them has been stymied by unstable training and the mismatch between dense softmax attention and spike-based communication.

The framework tackles both challenges head-on with three innovations: iPLMP, a Parallel LIF (Leaky Integrate-and-Fire) neuron with multistage learnable parameters plus a tailored P-STBP algorithm for stable deep training; SMSA, an attention-inspired spike-driven token-mixing module that replaces dense pairwise softmax with channel-wise spike co-activation and self-compensation; and SMoE, a spiking mixture-of-experts module for modality-aware fusion. On visual and multimodal benchmarks, SMM Transformer achieves accuracy competitive with conventional ANN baselines. More strikingly, under a standard MAC/AC arithmetic model, SMSA reduces estimated operator-level compute energy of the attention module by up to 97%, with whole-model profiling showing consistent, if more moderate, efficiency gains. The work shows that spiking architectures can be practical for multimodal tasks, potentially enabling on-device AI with dramatically lower power footprints.