Oyster-II uses RL to build safer, more helpful LLMs
Reinforcement learning replaces blanket refusals with constructive safety reasoning.
Large language models (LLMs) face a persistent trade-off between safety and helpfulness. Traditional refusal-based alignment often blocks legitimate user queries, frustrating users without addressing the underlying intent. The Oyster series tackles this with "constructive safety" – responding thoughtfully to sensitive queries instead of issuing blanket denials. Oyster-I used supervised fine-tuning (SFT) but suffered from poor safety generalization on out-of-distribution inputs and over-applied safety chain-of-thought reasoning to harmless questions, degrading user experience.
Oyster-II introduces a reinforcement learning (RL) framework built on a Zero-RL paradigm with multi-stage RL training. This approach dynamically adjusts the model’s reasoning, applying safety considerations only when needed and maintaining helpfulness on benign queries. Extensive benchmarks show Oyster-II comprehensively outperforms Qwen3-14B and Oyster-I across safety dimensions, with cross-scale results matching (and sometimes exceeding) much larger models like Qwen3-Max and Qwen3.5-397B. The work signals a shift from simple refusal to intelligent, context-aware safety alignment.
- Oyster-II uses multi-stage RL (Zero-RL paradigm) to overcome limitations of SFT-based constructive safety.
- Eliminates safety chain-of-thought over-generalization that hurt helpfulness on benign queries in Oyster-I.
- Outperforms Qwen3-14B and Oyster-I, matching Qwen3.5-397B across safety benchmarks.
Why It Matters
Smart safety alignment that distinguishes harmful from legitimate queries—without sacrificing helpfulness or scaling to huge models.