Korte et al. find dimensionality controls when modularity aids continual learning
High-dimensional representations make architecture irrelevant—but not in rich regimes.
Researchers Korte, Pedersen, Nisioti, and Risi studied how network architecture, task similarity, and representational dimensionality interact in continual learning. They compared a task-partitioned modular recurrent network with a single-module baseline, varying task similarity and weight initialization scale. In high-dimensional regimes, architecture had minimal impact. In lower-dimensional (rich) regimes, modular networks exhibited graded geometry: overlapping subspaces for similar tasks, partial orthogonalization for moderately dissimilar tasks, and stronger separation for dissimilar tasks. Single networks lacked this geometry. The study identifies representational dimensionality as a key variable governing when structural separation becomes functionally relevant.
- In high-dimensional regimes, single networks match modular ones, making architecture irrelevant.
- In low-dimensional (rich) regimes, modular networks show graded alignment based on task similarity.
- Single-module baselines lack adaptive geometry, causing interference across tasks.
Why It Matters
Guides engineers designing AI systems that must learn continuously without catastrophic forgetting.