Deep Neural Regression Collapse
New research proves deep regression models collapse into simple, low-rank structures across multiple layers.
Akshay Rangamani and Altay Unal's paper 'Deep Neural Regression Collapse' establishes that the Neural Collapse phenomenon—previously studied in classification tasks—also occurs systematically in deep regression models. Their key finding is that this structural simplification happens not just at the final output layer, but propagates through intermediate layers of the network. During training, the features in these 'collapsed' layers converge to lie within a low-dimensional subspace that directly corresponds to the target variable's dimension. Furthermore, the feature covariance matrices align with the target covariance, and the weights of the network layers align their input subspaces with this simplified feature space.
This work provides crucial theoretical insight into the internal representations learned by models tackling regression problems, like predicting continuous values. The researchers also investigated the role of weight decay regularization in inducing this collapse, showing it's often necessary for the phenomenon to occur. By proving models that exhibit Deep Neural Regression Collapse (NRC) successfully learn the intrinsic dimension of low-rank targets, the paper offers a more unified framework for understanding generalization in deep learning. This foundational discovery could inform the design of more efficient architectures, better regularization techniques, and improved training protocols for a wide range of AI applications beyond image classification, from scientific modeling to financial forecasting.
- Extends Neural Collapse theory from classification to regression, proving it occurs across multiple model layers (Deep NRC).
- Shows collapsed layer features align with target dimension subspace and covariances, simplifying internal representations.
- Identifies weight decay as a key factor in inducing collapse, helping models learn intrinsic low-rank target structures.
Why It Matters
Provides a mathematical blueprint for how AI models simplify complex tasks, guiding the development of more efficient and interpretable neural networks.