New 'Socratic' AI Asks Follow-Up Questions to Catch Poisoned Sources

Most AI chatbots you use don't answer from memory alone. They look things up first — a technique called RAG (letting AI search before it speaks). That's how they cite news articles or your company's documents. The problem: if the pages they find are wrong, planted, or tampered with, the AI confidently repeats the mistake. Researchers call this "evidence poisoning," and it's a real risk as more people trust chatbots for medical, legal, and money questions.

A new paper proposes a fix built on an old idea: ask a better question. Their system, Socrates-RAG, checks whether its sources actually agree. If two answers conflict, it doesn't just pick the more popular one. It pinpoints the single missing fact that would settle the argument, then runs one more search aimed specifically at that fact. Only then does it answer — or honestly admit it doesn't know.

In a controlled test of 48 scenarios, designed so the deciding evidence was never in the first batch of results, this approach lifted safe, correct answers from 79.2% to 93.8%. Unsafe answers fell to zero. Nearly all the gain came from two-step questions — the kind where you need fact A before you can find fact B, like finding a person's employer before their salary. Simple one-step questions were already solved by both versions.

The honest catch: 48 scenarios is a small, purpose-built test, not the messy open internet. The authors describe this as one isolated improvement, not proof that AI is now poison-proof. Still, the direction matters. Instead of bolting more filters onto a fixed pile of search results, the AI treats its own next question as part of the safety problem — which is a lot closer to how a careful human checks a rumor.