New theory reveals transformers use 'function vectors' for adaptive inference
Deep transformers implement distributed inference via function vectors, scaling context understanding layer by layer
A new paper by Ravin Raj and Gautam Reddy, published on arXiv, introduces a theoretical framework for understanding deep transformers as mean-field interacting systems that implement distributed inference. The authors propose that transformers use internal state representations called 'function vectors' to infer a latent context variable, with resolution increasing over layers. This model predicts a non-trivial relationship between non-Gaussian hierarchical structure in the context and the depth of the transformer, tested empirically with constrained linear attention transformers.
The findings demonstrate that feedforward blocks and depth allow transformers to perform a much richer class of in-context learning algorithms than previously described. This theory bridges a gap in understanding transformer internals, offering a mechanistic explanation for how they handle complex correlations among coupled variables. The work has implications for designing more efficient and interpretable transformer architectures, potentially leading to models that adaptively scale computation based on task complexity.
- Transformers model as mean-field interacting systems with distributed inference
- Function vectors enable hierarchical context inference across layers
- Feedforward blocks and depth enable richer in-context learning algorithms than previously known
Why It Matters
Explains transformer internals, enabling more efficient and interpretable AI architectures that adapt computation to task complexity.