Transformers aren't parrots: New paper says they build prompt-specific operations

Giunti and Garavaglia's paper, "The Transformer Revolution, Part 1," reframes how Transformers work during inference. Instead of merely regurgitating statistical patterns from training, they propose Transformers actively build parameterized transformations on the fly. Dubbed SIDPP, this mechanism treats token vectors as concepts to be transformed, while the transforming concepts are matrices and vectors generated from the input sequence itself. The key architectural innovation is output-weight interconnections: outputs from some neural networks directly determine the weights of others, alongside standard output-input connections. This lets the model create a bespoke computational path for each prompt.

The paper also quantifies this dynamic processing's significance, noting its contribution grows with prompt length and can equal or exceed static, training-derived processing—a property called strong prompt sensitivity. This implies that larger prompts don't just add context; they actively reshape the model's computational machinery. Practically, the framework offers new leverage for interpretability and control, since dynamic parameters could be observed and steered. It also hints at building smaller models that rely more on inference-time computation rather than massive static weights, addressing sustainability concerns.

Finally, the authors draw a bold parallel to neuroscience. They argue the human cortex has mechanisms capable of implementing SIDPP, suggesting human language processing may operate on a functionally similar architecture. This positions Transformers not as alien statistical machines but as approximations of a deeper cognitive principle, bridging AI research and brain science.