LLMs fail under stacked instructions—new benchmark reveals

A new arXiv paper, *Instruction Stacking Collapse*, exposes a critical flaw in how large language models (LLMs) handle complex, multi-constraint prompts. Researchers Atul Anand and Sourav Chattaraj evaluated three top-tier models—Claude Sonnet 4.6, GPT-5-mini, and Gemini 2.5 Flash—using a benchmark that stacks up to 24 simultaneous instructions (e.g., JSON output, tone control, word limits). The results are alarming: instruction-following rates plummet from ~96% to as low as 20% as constraints accumulate, with a structured pattern of conflicts emerging (e.g., JSON output clashes with 9 other constraints).

The paper introduces a training-free solution: an instruction compiler that rewrites stacked prompts in a single LLM call. This compiler significantly boosts weaker models like GPT-5-mini by up to +11 points in follow rate, while stronger models (already optimized for structure) see negligible improvements. The authors validated their findings with cluster-robust tests, ensuring the gains stem from the rewrite itself—not token reordering or measurement artifacts. All benchmark code, verifiers, and cached responses are open-sourced for full reproducibility.