MSE Loss Fails to Generate Superposition in Neural Networks – Researchers Warn

A new LessWrong post by philh and Linda Linsefors reveals that Mean Squared Error (MSE) loss is fundamentally unsuitable for training neural networks to encode features in superposition. The core claim: if you train a network with MSE loss to represent more features than it has neurons, the loss function does not reward superposition—in fact, it actively pushes toward non-superposition solutions. The authors present three lines of evidence: Linda’s own failed toy experiments, controlled toy model comparisons, and rigorous mathematical analysis. Notably, the well-known 'Toy Models of Superposition' paper avoided MSE by using a different effective loss function, and a 2026 paper on compressed computation under loss also confirmed that MSE fails while L2, L1, L_inf, and cross-entropy all work.

The mathematical reasoning centers on the fact that MSE loss (equivalent to L2 norm) encourages the network to spread error evenly across features, rather than exploiting the sparsity or structure needed for superposition. The authors caution that random initialization may sometimes produce apparent superposition, but it is not stable under MSE training. For researchers building toy models of superposition, the fix is simple: replace MSE with cross-entropy (if applicable) or with L1 loss (sum of absolute errors). This work serves as a practical warning against using MSE in such contexts, backed by clear experiments and theory.