Semantic Tool Discovery for Large Language Models: A Vector-Based Approach to MCP Tool Selection

A team of researchers has published a paper introducing a novel semantic architecture to solve a critical bottleneck in AI agent tool-calling. Currently, when Large Language Models (LLMs) like GPT-4 or Claude are connected to external tools via the emerging Model Context Protocol (MCP), they often receive descriptions of *all* available tools in their context window. This 'kitchen sink' approach wastes tokens, increases costs, reduces accuracy, and strains context limits, especially as MCP servers can host dozens to hundreds of tools.

The proposed solution uses a vector-based retrieval system, similar to RAG (retrieval-augmented generation) for documents, but for tools. It creates dense embeddings that capture the semantic meaning of each tool's capabilities. When an LLM agent needs to perform a task, the system compares the user's query against this index and dynamically injects only the 3-5 most relevant tool descriptions into the prompt. Benchmarks on 140 queries across 121 tools from 5 MCP servers show staggering efficiency: a 99.6% reduction in tool-related tokens, a 97.1% chance the correct tool is in the top 3 results (hit rate @ K=3), and a Mean Reciprocal Rank (MRR) of 0.91, indicating high ranking accuracy.

This architecture is a foundational upgrade for scalable AI agents. By moving from a static list to a dynamic, semantic discovery layer, it enables practical multi-agent systems and cross-organizational tool sharing without crippling overhead. The sub-100ms retrieval latency ensures it doesn't become a performance bottleneck. The work directly addresses the economic and technical barriers preventing developers from connecting LLMs to vast, real-world tool ecosystems.