Scaling of learning time for high dimensional inputs

A new theoretical paper by researcher Carlos Stein Brito, titled 'Scaling of learning time for high dimensional inputs,' reveals fundamental limitations in how neural networks learn from complex data. The research analyzes how learning time depends on input dimensionality using a Hebbian learning model performing independent component analysis. The key finding shows that learning times scale supralinearly with input dimensions, meaning they increase faster than linearly and become 'quickly prohibitive' for high-dimensional data. This explains the observed trade-off between model expressivity and learning efficiency in both artificial and biological neural networks.

The study demonstrates that for higher input dimensions, initial parameters have smaller learning gradients and larger learning times, with learning dynamics reducing to a unidimensional problem dependent only on initial conditions. Based on the geometry of high-dimensional spaces, the research outlines a new framework for analyzing learning dynamics and model complexity. These results have significant implications for designing optimal neural network architectures, suggesting that current approaches to scaling models with massive parameter counts may face fundamental efficiency barriers. The work provides theoretical grounding for why biological brains maintain relatively sparse connectivity despite processing high-dimensional sensory inputs.