Self-Verified Distillation lets LLMs improve without external data

A new paper from Stanford researchers Tony Lee and Percy Liang, titled 'Self-Verified Distillation: Your Language Model Is Secretly Its Own Synthetic Data Pipeline,' demonstrates that post-trained LLMs can further improve themselves using only unlabeled seed questions—no ground-truth answers, no external teachers, and no tool feedback. The method generates multiple candidate solutions per prompt, then filters them through a three-stage verification cascade: cycle-consistency, factuality, and correctness checks. Only solutions that pass all stages with unanimous judge votes are kept for training. The authors found that sampling more candidates and using a larger verification budget during data construction yields higher-quality self-curated data.

Experiments on Qwen3 models at 0.6B, 4B, and 8B scales show significant gains across math, science, and coding. For Qwen3-4B, held-out pass@1 improved by +16.7 points on math benchmarks (AIME26 and HMMT), +11.1 on science (GPQA Diamond and HLE), and +8.3 on coding (LCBv5 and LCBv6). Crucially, Self-Verified Distillation outperforms test-time-only compute scaling (UQ-TTC) while requiring only a single inference call at test time. This removes the need for expensive real-time verification and paves the way for fully autonomous model improvement at scale.