Free Decompression with Algebraic Spectral Curves
New technique infers spectral properties of giant models from tiny ones without expensive computation
A new paper on arXiv (2605.03634) tackles a core bottleneck in theoretical deep learning: applying random matrix theory to realistically large models. While spectral methods are powerful for understanding generalization, robustness, and scaling, they are computationally limited by matrix size—forcing researchers to work with toy models. The authors extend Free Decompression (FD), a technique for extrapolating spectral densities across matrix sizes, by recasting it as an evolution along algebraic spectral curves. This removes previous restrictive assumptions, allowing FD to handle spectral densities with multiple bulks, multiple scales, and point masses (atoms)—all characteristics of real-world data and modern architectures.
The framework is validated on Hessian and activation matrices from standard neural networks and large-scale diffusion models. By enabling accurate spectral inference from small proxy matrices, it opens the door to scaling theoretical analysis to models that are currently out of reach. The work bridges a gap between rigorous random matrix theory and practical ML deployment, offering a tool for predicting failure modes and generalization behavior without explicitly computing large matrices.
- Generalizes Free Decompression using algebraic spectral curves to handle multi-modal, multi-scale, and atomic spectral densities
- Enables inference of spectral properties of large matrices (e.g., Hessians of 100B+ parameter models) from small empirical ones
- Validated on neural network activation matrices and diffusion model Hessians, showing practical applicability to real-world ML
Why It Matters
Makes random matrix theory practical for analyzing large models, improving understanding of generalization and robustness at scale.