Self-distillation for rectified flow can provably improve teacher models
New proof shows optimal mixing of true and teacher velocities avoids model collapse.
A new arXiv paper by Roy, Mukherjee, and Patil tackles a critical issue in generative AI: how to improve a model using its own generated data without causing collapse. They focus on rectified flow (RF), a type of generative model that learns a velocity field to map noise to data. The authors propose optimal self-distillation (SD) where a student is trained on a weighted mixture of true RF velocities and the teacher's velocities.
For linear RF with ridge regularization, they prove an exact affine path identity and derive a closed-form optimal mixing coefficient. They show that the optimal coefficient follows a sign rule: positive mixing corrects under-regularized teachers, while negative mixing fixes over-regularized ones. Their method uses one-shot generalized cross-validation to avoid grid searches. Experiments on Gaussians, Gaussian mixtures, and image data show that optimal SD consistently improves velocity risk, mode recovery, and finite-step generation compared to both the teacher and pure distillation.
- Proves an exact affine path identity for linear rectified flow with ridge regularization.
- Derives a closed-form optimal mixing coefficient that follows a sign rule for teacher correction.
- Eliminates grid search via one-shot generalized cross-validation tuning.
Why It Matters
This gives a principled, math-backed way to self-improve generative models without risking collapse.