Biased Generalization in Diffusion Models

A team of researchers including Jerome Garnier-Brun and Luca Biggio has published a paper titled 'Biased Generalization in Diffusion Models' that challenges standard training practices for AI image generators like Stable Diffusion and DALL-E. The study identifies a critical phase called 'biased generalization' where models continue to improve on standard test metrics while actually memorizing and reproducing features from individual training samples. This occurs because diffusion models learn coarse structure early in training, but later stages resolve finer details in ways that become increasingly dependent on specific training examples.

The researchers demonstrated this phenomenon by training identical networks on disjoint datasets and measuring the mutual distances between generated samples. They found that after reaching the minimum test loss, models enter a phase where generated outputs show 'anomalously high proximity' to training data. Using a controlled hierarchical data model, they precisely characterized this onset and attributed it to the sequential nature of feature learning in deep networks. The implications are significant for privacy-critical applications where training data might include medical records, proprietary designs, or copyrighted material, suggesting that standard early stopping criteria may be insufficient to prevent data leakage.