IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs

A research team from Google and collaborating institutions has introduced IH-Challenge, a novel training dataset designed to solve a critical security flaw in large language models: instruction hierarchy (IH). IH defines how an LLM prioritizes conflicting instructions—such as a system prompt telling it to be harmless versus a user prompt trying to jailbreak it. The team found that robust IH behavior is difficult to train because failures are often confounded with general instruction-following issues, and models can learn shortcuts like refusing all requests. Their solution is a reinforcement learning dataset specifically crafted to teach models a concrete, trust-ordered policy for resolving these conflicts.

Fine-tuning OpenAI's GPT-5-Mini on the IH-Challenge dataset, combined with online adversarial example generation, yielded dramatic improvements. The model's IH robustness increased by an average of +10.0% across 16 diverse benchmarks, including in-distribution, out-of-distribution, and human red-teaming tests. Crucially, this security boost didn't come at the cost of capability; unsafe behavior plummeted from 6.6% to 0.7% while the model's helpfulness on general safety evaluations actually improved. The training effectively saturated an internal evaluation for static agentic prompt injection, a common attack vector. The team has released the IH-Challenge dataset publicly to spur further research into building more secure and trustworthy AI agents that can reliably follow developer intent.