Distributional KD: Multi-temperature views beat pointwise KL divergence

Token-level knowledge distillation (KD) traditionally matches teacher and student distributions using Kullback-Leibler divergence, which compares probabilities pointwise. Verbii and Lee show this misses crucial information: the KL gradient is blind to which wrong token receives probability mass. Their paper, 'The Distributional View of Knowledge Distillation,' represents the teacher not by one softened output but by a family of multi-temperature views—marginals along the annealing path of its logits—and trains the student against a geometry-aware aggregate of these views under an embedding-based ground cost.

They formalize a design space spanning mixtures, log-linear pooling, entropic Wasserstein barycenters, and a debiased Sinkhorn-divergence flagship, proving an exact collapse result: log-linear pooling of tempered views is equivalent to a single temperature, meaning only transport-based aggregation preserves multi-view information. On instruction-tuned Pythia models, experiments yield three empirical laws. First, the dispersion law: benefits grow with effective temperature dispersion, not the number of views. Second, dispersed views unlock the aggregation operator—barycenters separate from arithmetic mixtures exactly when transport-based aggregation begins to outperform averaging. Third, a two-regime picture governed by the ceiling gap Γ = PPL_SFT − PPL_T decides which loss is best; when the fine-tuned teacher barely beats a supervised student, a gentle transport objective wins but no KD beats supervised fine-tuning, while at a real ceiling the ranking inverts and the fidelity-generalization correlation flips.