Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

A team from Princeton University and Stanford University has introduced Self-Distillation Zero (SD-Zero), a novel post-training method that significantly improves the efficiency of aligning language models. The core innovation addresses a major bottleneck in AI training: the need for dense, token-by-token supervision, which is expensive to obtain from human experts or powerful teacher models. SD-Zero cleverly sidesteps this by training a single model to act as both a 'Generator' and a 'Reviser.' The Generator produces an initial response to a prompt, and the Reviser then conditions on that response and a simple binary reward (right/wrong) to produce an improved version.

This process effectively transforms a sparse binary signal into dense, token-level supervision. The Reviser's token distributions, conditioned on the Generator's flawed output and the reward, are then distilled back into the Generator itself through on-policy training. The researchers demonstrated SD-Zero's effectiveness on math and code reasoning benchmarks using the Qwen3-4B-Instruct and Olmo-3-7B-Instruct models, where it achieved at least a 10% performance improvement over the base models. It also outperformed strong baselines like Rejection Fine-Tuning (RFT) and GRPO under identical training budgets.

Key to SD-Zero's success are two emergent properties: 'token-level self-localization,' where the model learns to identify which specific tokens in its answer need revision based on the reward, and 'iterative self-evolution,' where the improving revision capability is continuously distilled back to enhance generation. This creates a virtuous cycle of self-improvement without external oversight. The method represents a major step toward more sample-efficient and autonomous AI training, potentially reducing reliance on costly human feedback or massive, proprietary teacher models for alignment.