Neural Dynamics Self-Attention for Spiking Transformers

A team of researchers has introduced a novel self-attention mechanism, LRF-Dyn, designed to overcome critical bottlenecks in Spiking Transformer models. Spiking Neural Networks (SNNs) are celebrated for their ultra-low power consumption, making them ideal for edge devices, but combining them with Transformer architectures has been problematic. Existing Spiking Transformers suffer from significantly worse performance than standard Artificial Neural Networks (ANNs) and require excessive memory during inference to store large attention matrices. The team's analysis pinpointed the Spiking Self-Attention (SSA) mechanism as the root cause of both issues.

Inspired by biology, the LRF-Dyn method introduces two key innovations. First, it incorporates a Localized Receptive Field (LRF) into the attention calculation, giving higher weight to neighboring pixels. This mimics how biological visual neurons work and strengthens the model's ability to understand local patterns, directly boosting performance. Second, and more crucially, the team reformulates the attention computation using the inherent "charge-fire-reset" dynamics of spiking neurons. This clever approximation completely eliminates the need to explicitly calculate and store the massive attention matrix, which is the primary source of memory overhead.

The results from extensive experiments on computer vision tasks confirm the dual benefits. Models using LRF-Dyn show substantially improved accuracy, narrowing the performance gap with conventional ANNs. Simultaneously, they achieve a dramatic reduction in memory consumption during inference. This breakthrough establishes LRF-Dyn as a foundational component for building practical, high-performance, and truly energy-efficient Spiking Transformers, paving the way for advanced AI on battery-powered devices like smartphones, drones, and sensors.