New cloud gateway scales LLM agents to 3,000+ tools with 98% recall

A team of researchers has proposed a scalable gateway system to solve the growing bottleneck of LLM agents accessing external tools via the Model Context Protocol (MCP). As agents increasingly rely on tool calling to act on real-world services, MCP has become the de facto interface. However, operating MCP at cloud scale introduces major hurdles: legacy services aren't natively compatible, rapid protocol evolution creates ongoing costs, and the number of tools an agent can use is sharply limited by context window size and inference overhead. The new system, detailed in arXiv paper 2607.15593, breaks the direct-connect model by inserting a gateway on the data plane that handles legacy integration, MCP variant consolidation, access control, tool recommendation, and session-aware routing.

The results are striking: a hybrid retrieval method delivers 98% Top-15 recall, and the system scales agent tool access to over 3,000 functions with high selection accuracy. Compared to baseline approaches, tool selection time is reduced by 8.9x and token usage by 23.8x, with low per-call overhead that remains stable under scale-out. For stateful MCP backends with multiple replicas, the gateway preserves session affinity without adding client complexity. The paper also shares deployment lessons from a production environment, suggesting practical paths for companies building agentic systems. This work directly addresses the tension between wanting agents to do more and the strict limits of today's LLM architectures.