Categorical Deep Learning: A Unified Theory for All Neural Architectures

The paper "Categorical Deep Learning is an Algebraic Theory of All Architectures" (Gavranović et al.) tackles the longstanding gap between deep learning's empirical success and its theoretical foundations. While architectures like transformers and CNNs evolved through scaling and experimentation, a unifying mathematical language remained elusive. The authors propose using category theory—specifically the universal algebra of monads valued in a 2-category of parametric maps—to elegantly subsume all neural network design patterns into a single framework. This allows representation of sequential computation, recurrence, and other operations through the compositionality of monads.

Beyond architecture theory, the article categorizes deep learning theory into three sub-domains: optimization theory (explaining optimizers like AdamW), functional theory (explaining generalization, grokking, double descent), and architecture theory (where CDL fits). CDL stands out by providing a formal bridge between abstract constraints and practical implementations, grounding itself in functional programming to make the theory directly applicable. The framework extends earlier work in Geometric Deep Learning and Topological Deep Learning, which focused on data with nontrivial internal structures, but now covers all architectures uniformly.