There Will Be a Scientific Theory of Deep Learning [R]

A new perspective paper, spearheaded by a lead author and featuring 14 contributors, makes a bold case that a scientific theory of deep learning is finally coalescing. The authors, who have dedicated years to studying deep learning exclusively, synthesize five distinct lines of evidence from recent research to support their claim: solvable toy settings that offer tractable insights, insightful limits that clarify model behavior, simple empirical laws governing training dynamics, theoretical frameworks for hyperparameters, and universal phenomena observed across architectures. They draw analogies to physics to contextualize these findings, hoping to inspire more rigorous scientific inquiry into how and why these vast learning systems function.

The paper is published on arXiv and has sparked discussion on social media, with an explanatory tweet thread from the lead author providing accessible commentary. By pulling together these disparate threads, the authors aim to galvanize the research community toward a unified understanding of deep learning, moving beyond empirical trial-and-error toward a principled science. This work could reshape how researchers approach model design, training, and interpretation, potentially leading to more predictable and efficient AI systems in the future.