AI Database Helpers Flunk Real-World Business Tests

Scientists created a new benchmark called ESQ-Bench to test how well AI models handle questions about enterprise databases — the kind companies use to store customer records, sales, and inventory. Unlike older tests that use simple, textbook-style data, this one uses realistic Oracle database setups with 465 tables and over 164,000 rows. They asked AI models in plain English, like "Show me all orders over $500 from last quarter," and checked if the resulting queries were correct.

The results are worrying for anyone thinking about letting AI run their data analysis. GPT-4o, one of the most powerful AI models, scored 79.8% accuracy on the easiest set of questions but fell to 57.2% on the hardest tier. Even more concerning: among queries that ran successfully, 73% to 99% returned results that looked reasonable but were actually logically wrong. The researchers call this "silent semantic divergence" — the AI gives you a confident answer that is subtly incorrect.

The study also compared other models. Claude Sonnet 4.6 outperformed GPT-4o on every difficulty level, reaching 87.4% on easy questions. Meanwhile, open-source models like Llama 3.2 performed terribly, managing only 13.3% accuracy overall. Clearly, there's a huge gap between the best commercial AI and free alternatives when it comes to navigating complex business data.

For businesses, this is a caution flag. Many companies are experimenting with "ask your database" features — where you can just type a question instead of hiring a data analyst. This research shows that in real-world conditions, these AI tools can be dangerously unreliable. They may miss key filters, confuse table relationships, or pull the wrong time range — and they don't warn you. Until these models improve, human oversight is essential.