Half of AI App Connectors Fail to Start, Study Finds

There's a growing standard called MCP (think of it as a universal plug socket that lets AI assistants connect to outside apps like calendars, databases, or file storage). Thousands of developers publish these little connector programs to a public directory. A researcher took a genuinely random sample — 400 of the 24,135 listed servers — and tried to connect to each one over the internet, exactly as a real user would.

Only 48.8% managed to complete the basic "hello, I'm here" handshake. Curated lists of hand-picked servers did better, at 66.7% — which means published overviews of this ecosystem have been quietly showing you the good half. The most common failure wasn't missing login details (13.3%); it was servers that simply never started at all (37.5%). The 195 that did run were surprisingly well-behaved: zero fatal formatting errors across 2,766 advertised tools. What they did skip were optional safety labels — missing 58.8% of the time on a random draw, versus 41.5% on the curated list.

The second half of the study is a warning about the scoreboards we use to rank AI helpers. The researcher compared the tool descriptions in real connectors against two widely used AI test sets. Real tools barely repeat themselves (2.8% near-duplicates, all inside the same project). One popular test set, BFCL v4, is 16.7% near-duplicate — and 68.8% of its raw rows are exact copy-paste repeats of the same name and description. The other, UltraTool, is 85.6% repeats. Without careful cleanup, anyone scoring an AI on these tests is partly measuring how often the same question appears.

So what? If you're paying for an AI assistant that promises to "connect to your tools," there's a real chance the connection just won't work, and the polished demos you've seen were likely cherry-picked. And when a company boasts that its model tops a tool-use leaderboard, that number deserves a raised eyebrow.