AI Chatbots Can't Tell When Their Sources Are Lying to Them
Your AI assistant may confidently give you a wrong answer when its sources are misleading.
Have you ever asked an AI assistant a question and it answered with total confidence — only to find out it was wrong? A new research paper reveals a sneaky reason why. These systems, called RAG (letting AI look things up), are designed to pull information from sources to answer you. But this study found that AI is much better at noticing when information is missing than when information is actively misleading.
The researcher created a test where the same questions were asked with four types of source material: helpful, slightly broken, missing, and deliberately misleading. The misleading versions tampered with a real source so it appeared to support a wrong answer. When source material was simply missing, the AI correctly said it didn't know. But when given a fake-but-plausible source, the AI happily answered 41.6% of those trick questions — and 63% of those wrong answers repeated the false information exactly.
Even trying to make the AI more careful didn't help much. Asking it to reason step-by-step barely improved things. One extra check reduced the wrong answers to 13.3%, but it also threw away lots of correct answers. Another check kept the correct answers but failed whenever the AI's own built-in knowledge agreed with the planted lie. The core problem: these systems check whether the source 'seems enough,' not whether it's actually true.
Why does this matter for you? AI assistants are increasingly used for research, medical questions, legal help, and shopping. A confident wrong answer from a misleading website can lead you astray — and you'd never know. This study shows that making AI trustworthy isn't just about giving it more data; it's about teaching it to question its sources.</p>
- AI assistants that look up information say 'I don't know' when sources are missing — but answer confidently when sources are misleading.
- In tests, the AI gave wrong answers 41.6% of the time on misleading questions, and repeated the false info word-for-word 63% of the time.
- Extra safety checks reduced errors but also discarded correct answers, showing the difficulty of balancing caution and usefulness.
Why It Matters
Whenever you trust an AI answer, it might be confidently wrong — especially if its source is misleading.