OPSA cuts safety tax in LLMs with on-policy self-distillation

Safety alignment in large language models often comes at a cost: reduced reasoning ability, a phenomenon known as the safety tax. Researchers from multiple institutions pinpoint a root cause — distributional mismatch from off-policy training, where models learn from safety demonstrations generated by humans or external models rather than their own outputs. To address this, they propose OPSA (On-Policy Self-Distillation). The model generates its own response trajectories and receives dense per-token KL divergence supervision from a frozen teacher copy of itself, conditioned on a privileged safety context. A novel metric called "teacher flip rate" measures how often the safety context converts an unsafe response into a safe one, enabling the system to discover contexts that activate latent safety reasoning rather than just mimicking safe patterns.

Tested on five model scales across R1-Distill and Qwen3 reasoning model families, OPSA consistently outperformed off-policy self-distillation and external-teacher distillation under matched data and full-parameter fine-tuning. The largest gains were on smaller models: +8.85 points on R1-Distill-1.5B and +5.49 points on Qwen3-0.6B. These improvements held across varying training-set sizes and adaptive jailbreak evaluations. Token-level analysis further revealed that OPSA concentrates gradient updates near early compliance-decision tokens, explaining how it improves safety while preserving general reasoning. This work offers a scalable path to reduce safety tax without sacrificing model performance.