RAMP framework exposes LLM agents' 80% failure rate in production workflows

A team of researchers led by Yipeng Ouyang has released RAMP (Runtime Assessing of Agentic Models in Production Systems), a new evaluation infrastructure designed to measure LLM agent performance in realistic, long-horizon software engineering tasks. Built on the YatCC integrated platform, RAMP introduces standardized orchestration and execution interfaces, along with compiler-construction workloads that involve serial dependencies, complex toolchain interactions, and staged recovery mechanisms. Unlike conventional isolated benchmarks, RAMP evaluates agents under conditions that mimic production environments—long execution chains, iterative feedback loops, and partial workflow failures.

When tested across 15 mainstream models, the results were stark: task completion rates progressively collapsed from 100% in the initial stage to only 20% in the final stage, with no model successfully completing the entire pipeline. Runtime analysis revealed systematic failure propagation and resource inefficiencies, with computational costs differing by up to three orders of magnitude among comparable models. The paper argues that benchmark scores alone are misleading and that continuous, runtime-observable assessment is essential for deploying reliable agentic systems.