Sparse neural networks obey asymmetric scaling laws with double-descent peaks

John Sous and Michael Winer from arXiv (stat.ML, 2026) present a groundbreaking theoretical analysis of scaling laws for neural networks with sparse features. Their model reveals that when activations are sparse, test loss is disproportionately influenced by rare coordinates never seen during training—a mechanism absent in dense networks. This creates a novel bottleneck that fundamentally alters scaling behavior. The loss curve shows a pronounced double-descent peak near the interpolation threshold (where parameters just fit the training data), governed by two separate scaling exponents: one for the underparameterized regime and another for the overparameterized regime. The gap between these exponents is determined by the degree of sparsity, challenging the standard power-law scaling assumptions.

Crucially, the authors derive a compute-optimal frontier that suggests allocating compute budgets to larger datasets rather than larger models yields better performance—a reversal of the typical scaling wisdom for dense networks. They also analyze gradient-descent dynamics, identifying a scaling law for the probability that fixed-step gradient descent becomes unstable. These sparsity-induced effects persist even with nonlinear activations, making the findings broadly relevant. This work has major implications for training sparse models (e.g., mixture-of-experts, pruning-based architectures) and suggests that optimal model scaling strategies may differ fundamentally from those used in dense transformers like GPT-4 or Llama 3.