Diffusion Models Collapse on Self-Training: New Paper Reveals Why

A new theoretical paper from researchers at the University of Cambridge and Google DeepMind (authors Naïl B. Khelifa, Richard E. Turner, and Ramji Venkataramanan) tackles the long-standing problem of model collapse in recursively trained generative models. The work, titled "Recursively Trained Diffusion Models: Limiting Collapse Distribution and Spectral Characterization," provides both the limiting distribution that iterative retraining converges to and its convergence rate. Crucially, the authors isolate a mechanism distinct from imperfect learning: even if a diffusion model has perfect score estimation and exact sampling, the early stopping of the reverse diffusion (necessary for numerical stability) drives a progressive drift away from the true data distribution. They prove that this recursion converges geometrically to a unique limiting distribution, which they characterize as an infinite mixture of increasingly Gaussian-smoothed versions of the data. Using a Hermite spectral decomposition, they show that recursive training acts as a low-pass filter, attenuating higher-order modes that encode fine non-Gaussian structure much more strongly than coarse modes. The paper proposes an "annealed truncation schedule" that progressively shrinks truncation times across retraining rounds, proving that any schedule converging to zero asymptotically eliminates recursive compounding. The authors also demonstrate robustness: even with discretization and score estimation errors, the learned distribution remains within a Wasserstein-2 ball around the ideal limit, with high-order errors contracting faster than low-order ones. The results are validated on synthetic Gaussian mixtures and CIFAR-10.