The paper that killed deep learning theory

LawrenceC's post on LessWrong argues that Zhang et al.'s 2016 paper 'Understanding deep learning requires rethinking generalization' effectively killed classical deep learning theory. The paper demonstrated that neural networks with tens of millions of parameters could perfectly memorize random training labels, yet still generalize well on real data—a direct contradiction of statistical learning theory's core tenets. Traditional complexity measures like VC dimension and Rademacher complexity predicted such models would overfit catastrophically, but they didn't. The finding shattered the dominant framework, which assumed generalization required a hypothesis class simple enough relative to data size. Instead, it suggested that the implicit biases of SGD and network architecture—not explicit capacity constraints—drive generalization. This forced researchers to develop new theories, like neural tangent kernels and simplicity biases, to explain why deep nets learn simple functions in practice.

Despite its impact, the paper didn't address all theoretical approaches; alternative perspectives like Bayesian inference and compression-based theories remained viable. But it marked a turning point, shifting focus from capacity-based bounds to understanding optimization dynamics and implicit regularization. The post notes that by 2026, with models like Claude 4.7 Opus consuming massive compute, the lesson remains relevant: theory must evolve to explain emergent behaviors of overparameterized systems. For professionals, this underscores that modern AI success relies on empirical insights over classical guarantees, urging a pragmatic approach to model design and evaluation.