Standard Transformers Achieve the Minimax Rate in Nonparametric Regression with $C^{s,\lambda}$ Targets

Researchers Yanming Lai and Defeng Sun have published a landmark theoretical paper demonstrating that standard Transformer architectures achieve minimax optimal rates in nonparametric regression for Hölder target functions. Published on arXiv (2602.20555), this 58-page work represents the first proof that Transformers can approximate Hölder functions C^{s,λ} with arbitrary precision under L^t distance, providing crucial mathematical validation for the architecture's success in large language models like GPT-4 and Claude 3, and computer vision systems like Vision Transformers.

The paper introduces two novel metrics—size tuple and dimension vector—that enable fine-grained characterization of Transformer structures, facilitating future research on generalization and optimization errors. As intermediate results, the researchers derived upper bounds for Transformers' Lipschitz constants and memorization capacity, findings that may influence model design and training methodologies. This theoretical foundation confirms why Transformers excel at learning complex patterns from data, potentially guiding more efficient architecture development and explaining their superior performance across AI domains from natural language processing to image recognition.