New Theory Shows Why Transformers Outperform CNNs and FCNs

A new paper by Peilin Liu and Ding-Xuan Zhou, published in Neural Computation and posted on arXiv, delivers a long-awaited mathematical foundation for why Transformer architectures work so well. Titled 'Generalization Analysis of Transformers in Distribution Regression', the work proposes a learning framework where distributions (rather than individual data points) are the inputs, directly linking to how natural language processing models handle variable-length sequences. The key innovation is the 'attention operator,' a formal mathematical representation of the attention mechanism that can compress entire distributions into compact function representations without any information loss. Using this operator, the authors prove that Transformers can learn functionals with significantly more complex structures than convolutional neural networks (CNNs) or fully connected networks.

Crucially, the paper also derives a generalization bound — a rigorous upper limit on how well the model will perform on unseen data — within the distribution regression setting. This bound provides the first theoretical support for several techniques that have made large language models (LLMs) so successful: prompt tuning, parameter-efficient fine-tuning, and efficient scaling. The authors show mathematically that these methods align with the structural advantages of the attention operator. For practitioners, this means the empirical success of strategies like fine-tuning only a fraction of parameters or scaling up models efficiently is not just luck — it's rooted in provable properties of the Transformer architecture. The work bridges a critical gap between theory and practice, offering a principled lens through which to design future AI systems.