New Chemistry Test Shows AI Answers Change With the Wording

AI chatbots are now being used for all sorts of things, including science. But a new study shows that when it comes to chemistry, these models are surprisingly easy to confuse. A team of researchers built a test called ChemDIRT that asks AI models chemistry questions in many different formats — using different wording, different ways of drawing molecules, and different types of tasks. The goal was to see if AI gives consistent answers.

What they found is worrying. When the same chemistry problem was phrased slightly differently, or when a molecule was represented in a different but equally valid way, the AI often gave different answers. Some models were accurate on simple questions but failed on complex ones. None of the models tested were reliable across all variations. This means that a person using AI for chemistry help might get a correct answer one time and a wrong answer another time, just because of how they phrased the question.

Why does this matter? Because companies and researchers are increasingly exploring AI to speed up drug discovery, design new materials, or even help students learn chemistry. If the AI gives inconsistent answers, it could lead to wasted time, bad decisions, or even dangerous outcomes in a lab. The researchers say their benchmark is not just a critique — it's a tool to help future AI models do better by testing them rigorously before they're trusted with real science.

The takeaway: AI is powerful, but still far from reliable in specialized fields like chemistry. This study is a reminder that you should always double-check AI's work, especially when the stakes are high.