Locality constraints shape RNN learning dynamics, ICML 2026 paper finds
Local approximations to gradient descent yield qualitatively distinct convergence behavior in linear RNNs.
Ezekiel Williams, Alexandre Payeur, and Guillaume Lajoie (Université de Montréal) applied dynamical systems theory to data-aligned linear recurrent neural networks (RNNs) to compare the learning dynamics of gradient-based algorithms under spatial and temporal locality constraints. Their study, accepted as a poster at ICML 2026, examines random feedback local online (RFLO) learning and truncated backpropagation through time (tBPTT) against standard BPTT. By decomposing network dynamics into orthogonal modes, the authors show that RFLO produces qualitatively distinct stationary solutions, stability properties, and convergence rates compared to BPTT and one-step tBPTT. Notably, RFLO-learned solutions are confined to low-rank perturbations of the initial parameters—a result that generalizes beyond the data-aligned setting.
This analytical framework provides a rigorous understanding of how biological plausibility constraints (e.g., only local information available to each synapse) alter learning outcomes. The findings have implications for neuromorphic computing hardware, where energy-efficient local learning rules are often required, and for neuroscience models of synaptic plasticity that must respect spatiotemporal locality. By revealing that RFLO's solution space is inherently low-rank, the paper suggests that biases toward simpler, lower-dimensional representations emerge naturally from locality constraints—a key insight for designing alternative optimization approaches for RNNs in both artificial and biological contexts.
- RFLO learning is theoretically restricted to low-rank perturbations of initial parameters, unlike BPTT.
- Dynamical systems analysis on data-aligned linear RNNs reveals qualitatively distinct stability and convergence for RFLO vs. BPTT.
- Accepted at ICML 2026 as a poster; extends implications to neuromorphic and biological learning models.
Why It Matters
This analytical result clarifies how biologically plausible learning rules inherently limit RNN expressivity, guiding future hardware and neuroscience research.