New Trick Stops AI Making Things Up When It Can't Find Answers

When you ask an AI assistant about your company's refund policy or a new regulation, it usually doesn't answer from memory. It first searches a stack of documents for relevant passages, then writes an answer based on what it found. This approach is called RAG (letting AI look things up). It works well — until the search comes back with nothing useful. In that moment, most systems have no idea anything went wrong, so they simply make something up. That's the polite definition of "hallucination."

The fix comes from an unexpected place: a Czech nuclear regulator. Because their data was too sensitive to send to outside AI companies, researchers had to find a way for the system to police itself locally. Their insight is that the search engine already hands over a kind of report card — a spread of relevance scores for the documents it found. When those scores look flat and shapeless, the search probably failed. When they cluster tightly around a few strong matches, it probably worked.

They tested this on more than 14,500 questions across eight image-heavy document collections. Their simple statistical check scored 0.856 on a standard accuracy measure, beating both older methods and a full local AI model used as a judge — which scored only 0.649 while running about 3,000 times slower and costing far more. The check takes roughly 2 milliseconds, faster than a blink.

The practical upside isn't just speed. A system that knows its search failed can say "I couldn't find this in your documents" instead of guessing — or flag the question for a human. For hospitals, banks, law firms, and government agencies that can't send private data to outside AI providers, this offers accuracy and privacy at once. The catch: on completely unfamiliar document types, accuracy drops from 0.856 to about 0.706, meaning it still needs occasional retuning.