Long-Horizon-Terminal-Bench: New benchmark reveals AI agents fail 85%+ of long tasks

A team of researchers from multiple institutions has released Long-Horizon-Terminal-Bench (LHTB), a demanding new benchmark designed to test AI agents on tasks that span minutes to hours. Unlike prior terminal benchmarks that reward only final outcomes, LHTB decomposes 46 tasks—spanning experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing—into fine-grained subtasks with dense intermediate rewards. This design allows evaluation to capture partial progress and open-ended workflows, stressing long-horizon planning, long-context management, and iterative debugging. On average, agents consumed 9.9M tokens per task, completed roughly 231 episodes, and executed for 85.3 minutes per run, making LHTB significantly more challenging than existing terminal benchmarks.

Results from 15 frontier models reveal stark limitations: even the strongest model achieved only 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0. The mean pass rate across all models was just 4.3% and 1.7% respectively. The researchers analyzed failure modes and error patterns, noting that agents frequently struggle with long-context retention, iterative debugging, and maintaining coherent plans over extended horizons. The benchmark is released publicly to support future progress on long-horizon terminal agents, emphasizing that current models have significant headroom for improvement in real-world, time-intensive tasks.