TRACE-ROUTER boosts agentic AI accuracy by 7+ points with 36% less latency

Researchers from Georgia Tech and IBM (Ritik Raj, Souvik Kundu, Sarbartha Banerjee, Dheemanth Joshi, Ishita Vohra, Tushar Krishna) have introduced TRACE-ROUTER, a novel task-level routing framework for agentic AI systems. Current LLM routers make independent per-call decisions, but agentic workflows run as long-horizon tasks where quality is only known at the end. This mismatch makes it impossible to attribute delayed reward to individual routing calls. TRACE-ROUTER solves this by routing entire tasks at admission using a contextual bandit algorithm, then locking all subsequent LLM calls to the chosen backend. It updates its policy using the terminal reward from the task, jointly optimizing for accuracy and latency without needing explicit task complexity estimation.

Across three agentic benchmarks, TRACE-ROUTER consistently achieves Pareto-dominant accuracy-latency trade-offs. On tau2-Bench, it delivers 7–8 accuracy points higher than a latency-matched interpolation between individual models. On Terminal-Bench, it hits 7.1 higher accuracy points than the strongest single model baseline while reducing latency by 36%. The framework is designed for enterprise AI deployments where cost-quality trade-offs are critical. By aligning routing with the actual unit of supervision—the task rather than each LLM call—TRACE-ROUTER enables smarter, adaptive model selection for complex agentic applications like customer support automation, code generation workflows, and multi-step research agents.