QIF neurons beat LIF in spike-based gradient descent training
New research shows QIF neurons yield smoother loss landscapes and better accuracy.
A new arXiv preprint from Carlo Wenig, Raoul-Martin Memmesheimer, and Christian Klos shows that quadratic integrate-and-fire (QIF) neurons significantly outperform the widely used leaky integrate-and-fire (LIF) neurons in spike-based gradient descent. The key issue with LIF neurons is that tiny parameter changes can cause spikes to appear or disappear, disrupting downstream activity and creating fragmented loss landscapes. This makes training unstable and can lead to permanently silent neurons.
In controlled experiments on the Spiking Heidelberg Digits dataset, the team performed a thorough hyperparameter search and found QIF networks consistently achieved higher accuracy. Visualization of loss and gradient landscapes confirmed that QIF neurons yield continuous, smooth surfaces, while LIF landscapes are fragmented and gradients erratic. The authors argue that adopting neuron models with continuous spiking dynamics—like QIF—can improve the reliability of gradient-based training for spiking neural networks, with implications for both neuromorphic computing and biological modeling.
- QIF neurons avoid spike discontinuities that cause unstable gradients in LIF neurons
- On Spiking Heidelberg Digits, QIF outperformed LIF after exhaustive hyperparameter tuning
- Loss landscape analysis shows QIF yields smooth, continuous gradients versus fragmented LIF landscapes
Why It Matters
A simple neuron model swap could make spiking neural networks easier to train and more reliable.