PACED: Distillation at the Frontier of Student Competence

A team of researchers has published a paper introducing PACED, a novel framework that fundamentally rethinks how knowledge distillation for large language models (LLMs) is performed. The core insight is that standard distillation is inefficient, wasting computational resources on problems a student model has already mastered (producing near-zero gradients) and problems far beyond its current capabilities (producing incoherent gradients that can erode existing skills). The researchers prove this waste is structurally inevitable, as the gradient signal-to-noise ratio vanishes at both extremes of a problem's pass rate.

To solve this, PACED introduces a principled method to focus training exclusively on the 'zone of proximal development'—the frontier of a student model's competence. It uses a Beta kernel weighting function, w(p) = p^α(1-p)^β, which is derived from the theoretical boundary-vanishing structure of distillation gradients. The framework is minimax-robust, meaning worst-case efficiency loss is minimal even under specification errors. In practical tests, PACED showed strong results in both standard teacher-student distillation using forward KL divergence and in self-distillation scenarios using reverse KL. The most effective approach was a two-stage schedule of forward-KL followed by reverse-KL, leading to substantial improvements on standard reasoning benchmarks. The method requires only student rollouts to estimate pass rates and is compatible with any existing model architecture.