BiSpikCLM: First binary spiking LLM cuts compute to 5% of normal

Large language models (LLMs) are incredibly powerful but notoriously energy-hungry. Spiking Neural Networks (SNNs) offer a brain-inspired, event-driven alternative that can slash power consumption—but previous spiking LLMs still relied on expensive floating-point matrix multiplications (MatMul) and nonlinearities. A new paper from researchers led by Sihang Guo tackles this head-on with BiSpikCLM, the first fully binary spiking MatMul-free causal language model.

BiSpikCLM introduces two key innovations. First, Softmax-Free Spiking Attention (SFSA) eliminates softmax and all floating-point operations in autoregressive language modeling. Second, Spike-Aware Alignment Distillation (SpAD) aligns an ANN teacher with the SNN student across embeddings, attention maps, intermediate features, and logits—enabling the student to match teacher performance using just 5.6% of the training tokens (for the 1.3B model). The result: BiSpikCLM achieves competitive natural language generation while consuming only 4.16%–5.87% of the computational cost of standard LLMs. This work establishes a viable path toward energy-efficient, brain-inspired NLP.