BM25 beats semantic embeddings for AI tool selection with 81% accuracy
BM25 achieved 81% top-1 accuracy vs 64% for embeddings in production tests
A developer building agents for a client with ~140 MCP-exposed tools switched from semantic embeddings to BM25 for runtime tool selection after discovering catastrophic failures in production. Semantic rankers (using text-embedding-3-small and cosine similarity) showed 64% top-1 accuracy on 200 query-tool pairs. The failure mode was insidious: wrong tools were confidently ranked first—for example, 'list the open issues for this repo' returned slack_search_messages before github_list_issues because the verb-noun embeddings diluted discriminative keywords like 'issues'. Tool descriptions are short (<50 tokens) and structurally similar, making them ill-suited for soft semantic ranking.
Replacing the ranker with BM25 over a flat-text projection of tool name + description + a walk of input_schema and output_schema (stripped of JSON structure) lifted accuracy to 81%. Hybrid (0.7 semantic + 0.3 BM25) underperformed at 78%, with semantic noise dragging down BM25's clean lexical signal. The developer notes that indexing schema property names like repo_id or branch adds critical discriminators. This contrasts with common RAG wisdom, where hybrid retrieval usually wins—but tool selection lives in a smaller, structured space where keyword matching excels.
- BM25 over tool name + description + schema walk achieves 81% top-1 accuracy, beating semantic embeddings' 64%.
- Hybrid retrieval (0.7 semantic + 0.3 BM25) scored 78%, worse than BM25 alone, contradicting typical RAG results.
- Indexing input schema property names (e.g., repo_id, branch) boosts BM25's discriminative power for tool selection.
Why It Matters
Developers building AI agents with many tools should prioritize keyword retrieval like BM25 over semantic embeddings for reliable tool routing.