[D] A mathematical proof from an anonymous Korean forum: The essence of Attention is fundamentally a d^2 problem, not n^2. (PDF included)
A viral math proof from a Korean forum could slash Transformer training costs from O(n^2) to O(nd^3).
An anonymous mathematical proof, originating from the Korean online community 'The Singularity Gallery,' has gone viral by challenging a fundamental assumption in large language model architecture. The author, who claims no affiliation with the LLM industry, presents the 'd^2 Pullback Theorem,' arguing that the Transformer's Attention mechanism is intrinsically a d^2-dimensional optimization problem, not the widely accepted n^2 problem tied to sequence length. This suggests the notorious O(n^2) computational bottleneck is an artifact of the softmax function, which artificially inflates the rank and destroys a more efficient Euclidean matching structure inherent in the data.
The paper proposes a concrete architectural change: replacing the standard softmax with a degree-2 polynomial kernel in a new 'CSQ Attention' layer. This modification purportedly preserves the necessary contrast for effective matching while stabilizing training and slashing computational complexity for both training and inference to O(nd^3), where 'd' is the embedding dimension. If validated by experts, this theoretical insight could provide the foundation for a new generation of Transformer variants that are radically more efficient, potentially unlocking longer context windows and faster model development without proportional increases in cost.
- The 'd^2 Pullback Theorem' mathematically proves Attention's core optimization is d^2-dimensional, not n^2.
- Proposes CSQ Attention, swapping softmax for a polynomial kernel to achieve O(nd^3) complexity.
- Anonymous author from a Korean forum seeks expert verification, claiming the finding could redefine Transformer architecture.
Why It Matters
If correct, this could dramatically reduce the compute cost of training and running state-of-the-art LLMs like GPT-4 and Llama 3.