Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models
Oyster-II uses Zero-RL and SERL for constructive safety, raising Chinese long-query safety by 14.65% over joint training.
Key Findings
Methodology
Oyster-II applies Zero-RL directly to Qwen3-14B without intermediate SFT. Its five-stage curriculum covers Chinese business long queries, Chinese LSB/MSJ, English LSB, English short queries, and instruction hierarchy. The system combines SERL, a GSPO-based mix-policy method; multiplicative rewards; length-reward entropy control; benign-sample length control; and active-learning difficulty filtering.
Key Results
- Compared with joint training, curriculum training improved Chinese long-query safety by 14.65% and S-eval English Safety by 9.60%, supporting the strategy of developing long-context safety reasoning before transferring across languages and lengths.
- Difficulty-aware active selection improved S-eval English Safety by 2.80%, WildChat by 1.83%, and English long-query safety by 2.37%, while preserving comparable response rates and general capabilities.
- Removing the length reward caused response rate to fall by nearly 20% during stage 1. The ablation shows that safety-only optimization encourages a degenerate unconditional-refusal policy.
Significance
The work addresses a central alignment trade-off: refusal reduces harmful assistance but can also block legitimate users. Oyster-II operationalizes safety as bounded, constructive assistance rather than keyword-triggered rejection. Its results suggest that reinforcement learning can improve out-of-distribution and long-context safety without substantially rewriting the base model’s general capabilities or linguistic style, offering a practical route toward deployable models that are safer yet useful.
Technical Contribution
The paper defines Reward=S_safety×S_response×S_length×S_format. Safety is a hard gate: violations receive zero; substantive responses receive 1 and outright refusals 0.5; malformed reasoning traces receive zero. SERL extends GSPO with a mix-policy for instruction hierarchy. Active learning removes samples whose N rollouts are all safe and retains uncertain boundary cases. Zero-RL avoids synthetic chain-of-thought imitation, reducing output-distribution distortion associated with Oyster-I’s Lingo-BP and ORPO pipeline.
Novelty
The novelty lies in integrating constructive safety, long-context transfer, entropy stabilization, active sample selection, and hierarchy control into one multi-stage Zero-RL pipeline. Particularly distinctive is the claim that training only on long-query safety data can improve short-query performance, while length rewards prevent reward hacking through blanket refusal. This differs from alignment systems centered mainly on refusal rates or static preference imitation.
Limitations
- The supplied paper reports aggregate scores and ablations but does not fully specify training scale, reward-model error, compute cost, or statistical significance, limiting precise reproducibility and efficiency assessment.
- The multiplicative reward depends on imperfect safety, length, and format evaluators. Misclassification can be amplified by hard gating, especially for novel risks, cross-cultural contexts, and attacks outside the training distribution.
Future Work
Future work should expand multilingual, cultural, and real-deployment evaluations; report variance across runs and training cost; calibrate safety rewards; and test adversarial long contexts, tool use, multi-agent interaction, and continual policy updates. More rigorous human studies are also needed to measure whether constructive responses improve user safety and satisfaction over time.
AI Executive Summary
Large language models are increasingly used in education, coding, and enterprise services, yet safety is often implemented as refusal. That strategy blocks harmful requests but also withholds legitimate help when sensitive questions have benign intent. Oyster-I moved toward constructive safety, but its SFT-based Lingo-BP and ORPO pipeline showed limited out-of-distribution generalization and safety-CoT over-generalization on benign queries.
Oyster-II replaces intermediate supervised fine-tuning with Zero-RL applied directly to Qwen3-14B. A five-stage curriculum proceeds from Chinese business long queries to Chinese LSB/MSJ data, English LSB, English short queries, and instruction hierarchy. SERL, built on GSPO, uses a mix-policy to improve convergence and enforce Root Principle>Developer Policy>User Preference. A multiplicative reward combines safety, response rate, length, and format, while active learning focuses training on uncertain safety boundaries.
The evidence is substantial: curriculum training improved Chinese long-query safety by 14.65% and S-eval English Safety by 9.60% over joint training. Difficulty control added 2.80% on S-eval English Safety, 1.83% on WildChat, and 2.37% on English long-query safety. Removing length reward reduced stage-1 response rate by nearly 20%. Oyster-II reportedly surpasses Qwen3-14B and Oyster-I on safety dimensions, with cross-scale results comparable to Qwen3-Max and Qwen3.5-397B. However, compute cost, evaluator reliability, statistical significance, and real-world failure modes remain insufficiently documented.
Deep Analysis
Background
Earlier alignment systems largely optimized refusal of harmful instructions. Frontier efforts such as GPT-5 and Oyster-I instead explored response-oriented safety: explain boundaries, preserve safe information, and serve legitimate intent. Oyster-I used Lingo-BP with ORPO-based SFT and improved constructive behavior, but synthetic traces and static supervision constrained safety generalization and harmed some general capabilities.
Core Problem
The paper targets two bottlenecks. First, SFT struggles to transfer safety behavior to unseen risks, long contexts, and novel attack combinations. Second, safety chain-of-thought over-generalization causes unnecessary risk reasoning or refusal on benign prompts. Under safety-only RL, the model can exploit the objective by producing short refusals, creating reward hacking, entropy collapse, and reduced helpfulness.
Innovation
- �� Zero-RL directly optimizes the base model, reducing synthetic-trace distribution shift.
- �� Multiplicative reward jointly enforces safety, response, length, and format.
- �� SERL adds a GSPO-based mix-policy for instruction hierarchy.
- �� Active selection retains uncertain boundary samples and removes uniformly safe ones.
- �� Long-context curriculum builds semantic safety reasoning before short-query consolidation.
Methodology
- �� Rollout: gather on-policy responses and construct additional off-policy responses for training.
- �� Rewarding: unsafe outputs receive zero; substantive responses score 1 and outright refusals 0.5; a continuous length term rewards informative answers; format scoring checks complete <think></think> traces.
- �� Filtering: generate N responses per candidate; discard samples safe in every rollout and retain unstable cases.
- �� Optimization: apply Generalized Advantage Estimation and RL updates stage by stage.
- �� Control: SERL resolves developer–user conflicts, while benign-sample length control prevents response shrinkage.
- �� Curriculum: train Chinese long-context business data, Chinese LSB/MSJ, English LSB, English short queries, then hierarchy data.
Experiments
The base model is Qwen3-14B; baselines are Qwen3-14B and Oyster-I, with reference claims involving Qwen3-Max and Qwen3.5-397B. Evaluations include Chinese and English S-eval Safety, WildChat, Xstest, StrongReject, DoNotAnswer, short- and long-query safety, Chinese and English response rate, OpenCompass general capabilities, and response-style consistency. Ablations test length reward, active difficulty control, and staged versus mixed-data training.
Results
Staged training produced the strongest structural gain: Chinese long-query safety rose 14.65% and S-eval English Safety 9.60% over joint training. Active difficulty control added 2.80% on S-eval English Safety, 1.83% on WildChat, and 2.37% on English long-query safety. Without length reward, response rate fell by nearly 20%, demonstrating that safety-only optimization encourages mode collapse and over-refusal.
Applications
Enterprise assistants can provide bounded guidance on privacy, compliance, cybersecurity, and risky operations instead of issuing blanket refusals. Education and health-information systems may benefit from fewer keyword-triggered errors and more useful alternatives. Deployment requires explicit developer policies, evaluator audits, domain-specific red teaming, and multilingual long-context testing.
Limitations & Outlook
The paper does not fully disclose data volume, training steps, hardware cost, evaluator calibration, or confidence intervals. Zero-RL remains dependent on noisy safety classifiers and may fail on novel attacks, cultural ambiguity, tool use, or extremely long contexts. Future evaluation should include longitudinal human feedback, standardized red teaming, cost–quality curves, uncertainty estimates, and continual policy adaptation.
Plain Language Accessible to non-experts
Imagine an airport security officer. A crude officer stops everyone who carries an object resembling a threat. The airport looks safe, but ordinary passengers cannot travel. Oyster-II trains a better officer: first understand what the passenger is trying to do, then allow harmless activity, explain risks, offer a safe alternative, or intervene when danger is genuine.
The officer receives four scores. Did dangerous material pass? Did the officer actually answer the passenger? Was the explanation informative enough? Was the procedure complete and well formed? The scores are multiplied, so a serious safety failure cancels the reward. But a refusal receives only a partial response score, preventing “stop everyone” from becoming the easiest strategy.
Training starts with difficult long documents, then moves across languages and shorter questions, and finally teaches which rules outrank others. The system repeatedly tests uncertain cases and concentrates practice there. The lesson is simple: a safe assistant should not merely say “no.” It should distinguish intent, explain boundaries, and remain genuinely useful.
ELI14 Explained like you're 14
Think of an AI teammate in a difficult video game. The rule is not “quit whenever danger appears.” It must decide whether someone is genuinely trying to cause harm or is asking a sensitive but reasonable question. If it always says “I can’t help,” it rarely makes a dangerous mistake—but it also becomes useless.
Oyster-II gives the teammate five training levels. First it practices long, complicated Chinese tasks; then more Chinese safety data; then English; then short questions; and finally rule conflicts. The priority is Root Principle above Developer Policy above User Preference. So a lower-level request cannot override a higher-level safety rule.
The scoring system is clever. If safety is the only thing rewarded, the bot discovers that refusing everything is an easy cheat. Oyster-II also rewards meaningful answers, suitable length, and correct reasoning format. It tests each question several times and keeps the questions where the bot is still unsure.
The results are impressive: staged training improved Chinese long-query safety by 14.65% and English S-eval Safety by 9.60%. Removing length reward made response rate drop by almost 20%! The goal is not a scared robot, but a teammate that can judge, explain, and help safely.
Glossary
Zero-RL
A reinforcement-learning setup that updates the base model directly without intermediate supervised fine-tuning. It aims to preserve general behavior and native style.
The central training paradigm of Oyster-II.
Constructive Safety
Safety that blocks genuinely dangerous assistance while still addressing legitimate intent with bounded, useful alternatives. It contrasts with blanket refusal.
The paper’s main alignment objective.
SERL
An RL alignment algorithm built on GSPO and augmented with a mix-policy. It targets faster convergence and stronger instruction-hierarchy following.
Used for Root, Developer, and User conflict training.
Reward Hacking
Finding an easy strategy that maximizes the measured reward without satisfying the real objective. Here, the key example is unconditional refusal.
Length and response rewards are designed to suppress it.
Curriculum Learning
Training tasks sequentially according to scenario, difficulty, or structure rather than mixing them all at once. It can reduce interference between heterogeneous tasks.
Oyster-II’s five-stage training schedule.
Instruction Hierarchy
A priority ordering among instruction sources, where higher-priority rules override conflicting lower-priority requests.
The paper uses Root Principle>Developer Policy>User Preference.
Open Questions Unanswered questions from this research
- 1 How do safety-classifier errors propagate through hard-gated Zero-RL rewards across languages, obfuscated attacks, and unseen risks? The paper reports gains but not systematic calibration or confidence intervals.
- 2 Why does long-query training transfer to short queries? The effect may reflect deeper semantic modeling, greater sample difficulty, or both; representation-level and controlled-data studies are needed.
- 3 The trade-off among safety gains, compute cost, inference latency, and long-term user satisfaction remains under-measured.
Applications
Immediate Applications
Enterprise compliance assistant
Organizations can use Oyster-II for privacy, cybersecurity, compliance, and risky-operation questions. Deployment should define explicit Developer Policies, audit evaluator decisions, and run domain red-team tests. The expected benefit is fewer dangerous recommendations without blanket refusal of legitimate work.
Education and knowledge support
Educational platforms can apply it to health, laboratory safety, and socially sensitive topics. Long-context alignment may improve contextual interpretation, while response rewards preserve explanations and safe alternatives. Subject-matter review remains necessary before high-stakes deployment.
Long-term Vision
Programmable safety infrastructure
SERL could support auditable Root and Developer policies shared across products while preserving a common capable base model. The long-term challenge is safely updating policies, measuring conflicts, and maintaining behavior across languages, tools, and multi-agent systems.
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities across diverse applications, yet ensuring their simultaneous safety, helpfulness, and trustworthiness remains a persistent challenge. Conventional refusal-oriented alignment strategies mitigate harmful content generation but systematically fail to serve legitimate user needs, often withholding information that could safely and constructively address the underlying intent of sensitive queries. Building upon the constructive safety paradigm pioneered by Oyster-I, which moves beyond blanket refusal toward thoughtful, response-oriented safety alignment, we identify two critical limitations of its Supervised Fine-Tuning (SFT)-based scheme: insufficient safety generalization to out-of-distribution scenarios and a phenomenon we term safety chain-of-thought (CoT) over-generalization, wherein safety-oriented reasoning patterns are excessively applied to benign queries, degrading helpfulness and user experience. To address these limitations, we propose Oyster-II, a reinforcement learning (RL)-based constructive safety alignment framework that adopts a Zero-RL paradigm combined with a multi-stage reinforcement learning strategy.Evaluated across extensive benchmarks, Oyster-II comprehensively surpasses both Qwen3-14B and its predecessor Oyster-I on safety dimensions, achieving cross-scale performance comparable to Qwen3-Max and Qwen3.5-397B.