New framework optimizes latency cost reliability in agentic AI workflows

Modern AI systems increasingly rely on multi-agent workflows where LLM-powered agents collaborate with conventional computational modules. A new paper from Ya-Ting Yang and Quanyan Zhu (New York University) tackles the core challenge of balancing three competing constraints: latency, reliability, and cost. The authors introduce performance models that capture how computational effort (e.g., reasoning tokens, output tokens) relates to output quality, using a parametric exponential reliability function for LLM agents. This allows precise modeling of the diminishing returns of throwing more compute at a problem.

The paper's main result is a water-filling token allocation policy for sequential workflows. This policy optimally allocates limited compute resources across agents to maximize overall reliability under latency and cost budgets. The authors also characterize optimal workflow reliability in terms of shadow prices, providing a practical way to evaluate tradeoffs. The framework is applicable to any sequential agentic workflow, from simple question-answering pipelines to complex research agents. It gives engineers a principled method to decide how many reasoning tokens to spend per agent step, balancing speed against accuracy and operational costs.