SLM-as-Router outperforms Claude Haiku 4.5 with 120ms latency

Specialized retrieval agents often surface higher-quality results than general-purpose search, but selecting the right agent for a given query remains a stubborn problem. Traditional routers infer query topic or intent, which fails when a topically aligned agent returns low-relevance results—the router has no feedback from retrieved content. A new arXiv paper (2608.00030) addresses this by training a small language model (SLM) to route based on actual retrieval performance, not just intent. The team used progressive supervised fine-tuning followed by reinforcement learning to jointly perform agent selection and generate structured parameters for downstream tool calls. A hierarchical reward function combines retrieval relevance with query-agent topic alignment, teaching the model when to redirect queries away from specialized agents despite surface-level topical overlap.

On a targeted subset of agent-query mismatches, the trained SLM achieves an NDCG@10 of 0.918, compared to 0.539 and 0.490 for Amazon Nova Lite and Claude Haiku 4.5, which route on intent alone. Overall, the model reaches a mean NDCG@10 of 0.771—improving on Nova Lite by +0.177 and on Haiku by +0.219—with a mean selection latency of just 120.1ms, an 82.4% reduction versus Nova Lite. These results show that compact models can act as efficient, accurate routers in multi-agent retrieval systems, learning task-dependent agent suitability from observed performance rather than relying on static intent mapping. This points toward cheaper, faster orchestration of agent networks, potentially reducing the need for expensive large-model calls in query routing.