Single-neuron experts make transformers more efficient and interpretable
What if each expert in a mixture-of-experts model was just one linear neuron?
A new paper by Simon Schug proposes a radical simplification of mixture-of-experts (MoE) architectures. Instead of large, dense experts, 'sgatlin' (sparsely gated linear neurons) shrinks each expert to a single neuron that uses no nonlinear activation. By selecting only a tiny fraction of a vast pool of such linear neurons, the model achieves extreme sparsity without proportional compute costs. Counterintuitively, removing nonlinearity is key: it allows the model to maintain or even improve performance while gaining interpretability.
In an isoflop comparison across different compute budgets, replacing all transformer feedforward layers with sgatlin improved perplexity in language models. The linearity makes the feedforward circuits directly interpretable: a small-scale case study showed that neurons form semantically structured clusters and are causally implicated in factual recall—all without training additional replacement models. The work points toward compute-efficient and interpretable transformer layers, potentially scaling to larger models while keeping inference costs low.
- Each expert reduced to a single linear neuron, removing nonlinearity entirely.
- Spatially-gated linear neurons (sgatlin) applied to transformer feedforward layers.
- Improves perplexity in isoflop comparisons across multiple compute budgets.
- Interpretable circuits: neurons cluster semantically and correlate with factual recall.
Why It Matters
Makes large MoE models more compute-efficient and transparent, enabling interpretability without extra models.