[R] Learning State-Tracking from Code Using Linear RNNs
Transformers fail where simple linear RNNs succeed at a core programming task.
Deep Dive
New research shows linear RNNs outperform Transformers on state-tracking tasks when framed as code execution. The study converted permutation composition problems into REPL code traces with interleaved prints and variable transformations. While linear RNNs excelled at tracking program state in this setting, Transformers consistently failed. The work also reveals a key difficulty: actions in code are often not fully observable, creating challenges even for specialized architectures.
Why It Matters
This challenges the Transformer's dominance and could reshape how we build models that reason about code execution.