Relay-Bench tests LLMs on multi-domain chains; GPT-5.5 scores 43.3%

Liam Swayne’s new benchmark, Relay-Bench, measures how well LLMs can chain together reasoning across multiple distinct domains in a single prompt. Each problem is a composite of 2 to 13 subproblems drawn from areas like visual reasoning, coding, math, information extraction, web search, problem-solving, general knowledge, and data analysis. The test set is unsaturated, meaning no model has solved it completely — the leader, GPT-5.5 (xHigh), only achieves 43.3% accuracy. Problems are deliberately made harder by encoding prompts in complex ways and adding ‘context bloat’ to distract models. No input/output are multimodal, but models are explicitly allowed to use code execution, web searches, and all available tools. The goal is to evaluate whether LLMs can maintain coherent reasoning across shifting domains, a skill that is critical for real-world applications like multi-step research or troubleshooting.

Relay-Bench stands out because it moves beyond single-domain benchmarks (like GSM8K for math or HumanEval for code) by demanding that a model seamlessly integrate different reasoning styles within one response. The 43.3% score from GPT-5.5 highlights a major gap: even frontier models struggle to sustain accuracy when subproblems change domains. The paper includes 21 pages and 7 figures analyzing failure patterns, such as models ‘forgetting’ the first subproblem’s output while solving the last. By forcing models to combine coding, math, and knowledge retrieval, Relay-Bench exposes weaknesses that simpler benchmarks miss. For practitioners, this suggests that multi-domain reasoning chains remain a fundamental challenge, and that tool use alone does not guarantee success. The benchmark is available for researchers to test their own models, offering a new stress test for the next generation of LLMs.