A theory of learning data statistics in diffusion models, from easy to hard

A team of researchers has published a new theoretical framework that demystifies how diffusion models, the powerful AI behind image generators like DALL-E 3 and Stable Diffusion, actually learn from data. The paper, titled 'A theory of learning data statistics in diffusion models, from easy to hard,' introduces a key metric called the 'diffusion information exponent.' This invariant proves that these models exhibit a 'distributional simplicity bias,' meaning they rapidly learn basic, pair-wise correlations in data (like simple shapes and colors) with linear sample complexity. Only after mastering these fundamentals do they begin the much harder task of learning complex, higher-order statistics.

The researchers validated their theory using a controlled data model, showing that learning intricate patterns—such as the fourth cumulant, which governs fine details and textures—requires at least a cubic increase in training data. However, they also proved this complexity barrier can be bypassed. If the simple and complex statistics share an underlying latent structure, the model can learn the hard parts with linear efficiency. This work provides the first rigorous mathematical explanation for the observed training behavior of modern generative AI, moving beyond empirical observation to a provable theory of learning progression.