TEFormer: Spiking Transformer with Bidirectional Temporal Fusion Beats Benchmarks

A new paper accepted at ICML 2026 introduces TEFormer, the first Spiking Transformer framework to achieve structured bidirectional temporal enhancement. Drawing inspiration from feedforward feedback modulation in the human visual pathway, TEFormer decouples temporal modeling across its core components: a lightweight, hyperparameter-free forward temporal fusion mechanism in the attention module enables fully parallel computation, while a backward gated recurrent structure in the MLP aggregates temporal information in reverse order to reinforce temporal consistency. This design allows the model to exploit spatiotemporal dependencies more effectively than prior Spiking Transformers, which lacked a principled mechanism for temporal fusion.

The researchers conducted extensive experiments across diverse benchmarks, showing that TEFormer consistently and significantly outperforms strong SNN and Spiking Transformer baselines. In a first systematic evaluation of Spiking Transformers under different neural encoding schemes, TEFormer's performance gains remained stable across encoding choices—indicating that improved temporal modeling directly translates into reliable accuracy improvements regardless of spiking representation. The model is fully open-source with code available on GitHub, marking a step toward energy-efficient, biologically plausible sequence modeling for real-world applications like video processing and neuromorphic computing.