New Distillation Method Beats Teachers Without Original Training Data

A new theoretical paper from researchers Hien Dang, Pratik Patil, and Alessandro Rinaldo tackles a practical machine learning problem: how to improve a trained model when the original labeled training data is no longer available. Their answer is a technique called prediction-only distillation, studied through a 'fresh-X prediction-mixed scheme.'

In this scheme, a pure-distilled student model is trained on fresh, unlabeled covariates pseudo-labeled by the original teacher model. The final predictor is an affine combination of teacher and student predictions. For ridge regression under proportional asymptotics, the authors derive deterministic equivalents for the optimally mixed prediction risk, showing it is strictly smaller than the teacher's risk for almost every regularization pair. This holds even when the fresh covariates come from a different distribution than the training data, including isotropic covariance. For binary logistic regression, mixing predictions outperforms both the teacher and the pure-distilled classifier alone.

The key insight is that the optimal mixing weight cannot be identified from unlabeled data alone, but can be consistently estimated using a small independent labeled calibration set in a single post-training step—no additional model fitting required. The paper spans 73 pages with 21 figures and 12 tables, providing rigorous theoretical guarantees.