Fully Looped Transformer stabilizes 12-loop training, boosts performance 13.2%

A team of researchers from Jilin University and other institutions has introduced the Fully Looped Transformer (FLT), a parameter-free modification to Looped Transformers that solves two key sources of training instability: gradient oscillation and residual explosion. Looped Transformers reuse the same blocks iteratively, offering a way to trade compute for performance without increasing model size. However, they collapse when loop iterations exceed a few steps. FLT's Fully Looped Architecture distributes inter-loop signals across all layers, mitigating residual explosion, while Attention Injection reuses the existing attention block to suppress gradient oscillation. Experiments show FLT trains stably up to 12 loop iterations, whereas other looped models fail. Even at fewer iterations, FLT improves average downstream performance by up to 13.2%.

By decoupling parameter count from compute, FLT allows practitioners to adjust loop iterations at inference time, providing a natural mechanism for balancing performance and test-time compute budgets. This makes FLT a practical alternative to scaling model size for improving performance, especially in resource-constrained settings. The modifications are simple and require no additional training overhead, relying only on architectural changes to the existing transformer. The paper provides evidence that FLT not only stabilizes training but also enhances generalization across multiple benchmarks, suggesting that iterative computation can be a reliable path to better performance without the cost of larger models.