W2S-OPD: Weak models can now supercharge stronger LLMs
New technique uses contrast pairs of weak models to improve strong student without larger teacher.
On-policy distillation (OPD) typically requires a teacher at least as capable as the student, limiting its use at the frontier where no larger model exists. To overcome this, the authors propose Weak-to-Strong On-Policy Distillation (W2S-OPD), which leverages contrast pairs of weak models to create a proxy teacher in logit space. The proxy teacher is formed by adding the logit difference between a positive and a negative model (both smaller than the student) to the student's base logits. This isolates a capability direction (e.g., skill from RL, scale, or correct reasoning) while keeping the teacher distributionally adjacent to the student. The student then minimizes per-token reverse KL divergence on its own rollouts, aligning to the proxy teacher without needing a larger model.
W2S-OPD is instantiated with three types of contrast pairs: (i) a post-RL expert vs. its pre-RL initialization, isolating the skill RL instills; (ii) a larger vs. a smaller base model, isolating capability from scale; and (iii) a small base model with correct vs. wrong hints, isolating instance-level direction. Experiments across four math benchmarks (e.g., GSM8K, MATH) and three code benchmarks (e.g., HumanEval, MBPP) show that W2S-OPD consistently outperforms standard OPD, enabling the student to surpass the domain teacher. Notably, the student continues to improve even when every supervision source is weaker. Analysis reveals that different contrasts yield distinct signals—post-RL and hint contrasts emphasize reasoning frameworks, while scale contrast emphasizes solving procedures. The method opens a path for frontier models to self-improve using cheap, weak models.
- W2S-OPD constructs a proxy teacher from the logit difference of a positive-negative contrast pair (e.g., post-RL vs. pre-RL) to isolate a capability direction.
- Outperforms standard OPD across 4 math and 3 code benchmarks, enabling the student to surpass the domain teacher.
- Student LLM improves even when all supervision sources are weaker, using cheap weak models for distillation.
Why It Matters
Enables frontier LLMs to keep improving without larger teachers, democratizing AI advancement with cheap weak models.