Frozen random CNNs spontaneously develop extreme sparsity in deep RL

A striking new preprint from Scott Norton on arXiv (cs.LG/2607.26059) reports that deep reinforcement learning agents trained with frozen, randomly initialized CNN feature extractors spontaneously develop extremely sparse fully-connected representations, despite no sparsity-inducing objective. In the first fully-connected layer (FC1, 3,136→64), agents compress task-relevant information through as few as 1-3 out of 64 neurons for deterministic Pong (5-11 for stochastic Pong), while trainable CNNs activate 55-64 neurons under matched conditions. This sparsity scales with task complexity: 1-11 for Pong, 19-26 for Breakout, and ~42 for Space Invaders. Width-scaling confirms the pattern reflects task structure rather than a fixed capacity fraction. Interestingly, within-game scaling emerges: three identical Pong seeds produce 5, 7, and 11 active neurons, with the 5-neuron seed plateauing at +14 reward while the others reach expert performance (+18.4, +18.7), suggesting the random projection's usable dimensionality bounds achievable performance.

Ablation experiments confirm the necessity of these active neurons: removing them crashes performance across two PPO implementations and four games. The information bottleneck commits early: a sweep shows the active set locks by 15-30M steps, while reward turns positive 35-105M steps later. In Breakout, frozen and trainable CNNs reach competitive rewards via structurally different bottlenecks: frozen agents use 17-25 active neurons (participation ratio ~10-14), while trainable agents use 51 (participation ratio ~3.6). The authors conclude that wherever input dimensionality dwarfs intrinsic task dimensionality, gradient descent on a frozen random projection may reveal the effective rank of the underlying problem without explicit sparsity machinery. This work challenges the assumption that trainable feature extractors are essential for deep RL and opens new avenues for efficient, resource-constrained learning.