AI Agents Keep Calling Tools That Don't Exist, Study Finds
Your AI helper may be inventing buttons that were never there — and bigger models don't help.
When you ask an AI assistant to book a hotel or send an invoice, it doesn't do that itself. It calls a "tool" — a small piece of software that actually performs the task. A new paper finds these assistants sometimes call tools that were never built at all, inventing names and details out of thin air. Think of a new employee confidently dialing an extension that doesn't exist, then reporting that the call went through.
The paper's core finding is that current safety measures can't catch this. Existing defenses either help the AI pick the right tool from a known list, or block what it's allowed to do with real tools. Neither one checks whether the tool exists in the first place. If the AI invents a tool, there's no rule that was broken — nothing to reject. The author argues the check must come first, like a bouncer verifying your name is on the guest list before anyone cares what you're wearing.
The numbers are striking. Across ten publicly available AI models, researchers counted 322 genuine fake-tool calls. These happened far more often when the AI was free to write raw code-like requests (34 cases) than when it used a structured, guided format (3 cases). Most surprising: size doesn't help. A model with 675 billion parameters — one of the biggest around — made the same mistakes as models roughly a hundred times smaller.
The problem gets worse with the Model Context Protocol, an emerging standard for plugging multiple tool servers into one AI. Merging those servers into a single shared list creates name collisions and hidden overlaps that no single directory can describe. On this combined setup, the team measured 154 more fake calls — including from top-tier models that had been completely clean before. To help, they released a versioned benchmark (a shared test set) so anyone building a fix can prove it works.
- AI assistants sometimes call software tools that don't exist — and current safety checks literally cannot catch it, because no rule was broken.
- Researchers measured 322 fake tool calls across ten AI models; a 675-billion-parameter giant performed no better than a tiny 7-8 billion one.
- Combining multiple tool sources under one standard (MCP) added 154 more fake calls, even from top models that were previously flawless.
Why It Matters
Fake tool calls could mean wrong bookings, failed payments, or silent errors — and paying more for a bigger AI won't prevent them.