There Will Be a Scientific Theory of Deep Learning

A team of 14 researchers from leading institutions has published a landmark 41-page paper arguing that a scientific theory of deep learning is coalescing. Dubbed 'learning mechanics,' this emerging framework seeks to characterize the training process, hidden representations, final weights, and performance of neural networks through five key research strands: solvable idealized settings that provide intuition for learning dynamics, tractable limits revealing fundamental phenomena, simple mathematical laws capturing macroscopic observables, theories of hyperparameters that simplify training, and universal behaviors shared across systems.

The paper positions learning mechanics as distinct from statistical or information-theoretic approaches, emphasizing falsifiable quantitative predictions and coarse aggregate statistics. The authors highlight a symbiotic relationship with mechanistic interpretability and address common skepticism about the possibility or importance of such a theory. They conclude with open directions and beginner advice, hosted at an accompanying website. This work signals a shift toward a more rigorous, physics-like understanding of deep learning, potentially enabling better model design, hyperparameter tuning, and predictability.