New paper disproves Fourier Alignment in single-neuron modular addition
A single ReLU neuron fails to learn modular addition under common training setups, per arXiv:2608.04451.
Researcher Gautam Neelakantan Memana published a paper on arXiv disproving Fourier Alignment in single-neuron modular addition. The study constructs a counterexample where a ReLU neuron, initially active, becomes inactive during training and remains frozen. The neuron's Fourier energy becomes evenly distributed across all nonzero frequencies, contradicting the assumption that single-frequency alignment is a general outcome of training.
The paper’s appendix, prepared by OpenAI’s GPT-5.6 Sol, strengthens the counterexample by showing the same failure occurs across Clarke trajectories, smooth dead-zone approximations of ReLU, and fixed-step full-batch gradient descent. These findings challenge the generality of Fourier Alignment in neural network training, particularly for modular addition tasks.
- Researcher Gautam Neelakantan Memana disproves Fourier Alignment in single-neuron modular addition via a counterexample on arXiv:2608.04451.
- A ReLU neuron becomes inactive during training, with Fourier energy evenly distributed, challenging neural alignment assumptions.
- OpenAI’s GPT-5.6 Sol verifies the counterexample generalizes across training conditions and neuron approximations.
Why It Matters
Challenges core assumptions in neural network training efficiency and interpretability for modular arithmetic tasks.