Subcritical Signal Propagation at Initialization in Normalization-Free Transformers

A new theoretical paper by researcher Sergey Alekseev provides a crucial mathematical framework for understanding why certain transformer architectures fail during training. The study introduces the Averaged Partial Jacobian Norm (APJN) as a measure of gradient amplification across layers and extends its analysis to transformers with bidirectional attention. The core finding is a clear 'criticality' picture: standard pre-LayerNorm transformers show controlled, power-law APJN growth, while normalization-free variants using elementwise nonlinearities like tanh exhibit problematic stretched-exponential growth.

This stretched-exponential behavior classifies architectures like Dynamic Tanh (DyT) and Dynamic erf (Derf) transformers as 'subcritical.' The theory mathematically explains the empirical observation that these models are notoriously sensitive to initialization and optimization hyperparameters, often requiring extensive tuning for stable training. The analysis, validated against deep vision transformers, provides a predictive tool for architects, allowing them to foresee training instability issues before running costly experiments. This work bridges a gap between the empirical engineering of transformers and rigorous theoretical analysis of their training dynamics.