InflationAgent router cuts token costs 31% while boosting accuracy

A new arXiv paper (2608.13571) from Heming Fu and colleagues tackles a blind spot in LLM routing: token inflation. Agentic systems that retry failed queries consume extra tokens each attempt, so the real cost of a workflow can be far higher than a single-call price implies. In tests, this inflation reached 4.25x for a 7B model on multi-hop question answering—meaning naive routers like FrugalGPT that optimize per-call cost may underestimate true spend by more than 2x on hard tasks.

To fix this, the researchers built InflationAgent, a four-stage router. It first measures token inflation systemically across model tiers and task types. Then it introduces CoT Branching Entropy (CBE), a pre-execution difficulty metric computed entirely from local inference, which flags likely-inflated runs with an AUROC of 0.887. Next, it ranks models by a Semantic Exchange Rate (SER)—expected accuracy divided by predicted true cost—and finally applies a fresh-escalation policy that discards failed reasoning chains before sending the query to a stronger model. The design is validated by experiments on GSM8K: InflationAgent achieved 94.7% accuracy versus 91.0% for FrugalGPT under a fixed budget, while consuming 31% fewer tokens. Notably, forwarding a failed chain to GPT-4o directly reduced accuracy by up to 34.8 percentage points, confirming that stale reasoning paths can actively poison stronger models. The result is a routing strategy that optimizes for actual workflow cost, not just sticker price.