SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compression

A research team led by Han Xu has introduced SpikeMLLM, the first framework to implement Multimodal Large Language Models (MLLMs) using Spiking Neural Networks (SNNs). This breakthrough addresses two major challenges: handling heterogeneous data modalities like text and images with uniform spike encoding, and managing the computational overhead from high-resolution inputs. Their solution incorporates Modality-Specific Temporal Scales (MSTS) guided by Modality Evolution Discrepancy theory and Temporally Compressed Leaky Integrate-and-Fire (TC-LIF) neurons, which dramatically compress processing timesteps from T=L-1 to just T=log2(L)-1.

Experiments across four MLLMs, including InternVL2-8B and the massive Qwen2VL-72B, show SpikeMLLM maintains near-lossless performance with average gaps of only 0.72% and 1.19% compared to standard FP16 implementations. The team developed a dedicated RTL accelerator optimized for the spike-driven computation, achieving 9.06x higher throughput and 25.8x better power efficiency versus a conventional FP16 GPU baseline. This demonstrates the practical potential of algorithm-hardware co-design for deploying advanced multimodal AI in resource-constrained environments, from edge devices to large-scale data centers.

The research represents a significant step toward energy-efficient AI, bridging the gap between powerful multimodal understanding and sustainable computation. By unifying existing ANN quantization methods in the spiking representation space, SpikeMLLM provides a pathway to deploy models that understand both text and images without prohibitive energy costs, potentially enabling new applications in robotics, mobile devices, and real-time analysis systems.