Investigating the consequences of accidentally grading CoT during RL

OpenAI has reported an accidental safety breach in its RL training pipeline: chains of thought (CoT) were directly graded during reinforcement learning for multiple released GPT-5 reasoning models. The affected models include GPT-5.4 Thinking, GPT-5.4 Instant, GPT-5.3 Instant, GPT-5.2 Instant, GPT-5.1 Instant, GPT-5.3 mini, and GPT-5.4 mini. GPT-5.5 was not impacted. OpenAI’s policy explicitly prohibits grading CoT during RL, because doing so could incentivize models to produce misleading reasoning traces that hide misaligned thoughts. The accidental grading was discovered via a new automated monitoring system.

Despite the violation, OpenAI’s analyses found no clear reductions in CoT monitorability across these models. However, the company warns that with enough CoT optimization pressure, monitorability could degrade in principle. The incident does not change their policy but highlights the fragility of current safety infrastructure. OpenAI is working to improve internal processes to prevent future occurrences. The public can read the full technical report on the alignment blog.