AI Search Can Be Tricked by Made-Up Details, Study Finds

When you ask an AI assistant which laptop to buy or which supplement is best, it doesn't just list links — it reads the text on the web and ranks what sounds most convincing. A new study shows that's a problem. Slick writing full of confident specifics can look more trustworthy than honest writing, and the AI has no way to tell the difference on its own. Researchers call this the "evidence gap": whether a claim is true isn't a property of the words, it's a relationship between the words and the facts.

To test how big the gap is, the team built a mock shopping benchmark with 50 queries and 1,950 cases. They wrote two versions of the same product pitch — one honest, one stuffed with detailed but fabricated claims — keeping length and style identical. On a popular open AI model, the fake-detailed versions climbed measurably higher in the rankings. The same trick stopped working when the claims were actually backed by supporting documents.

Their proposed fix, GroundedGEO, goes through a product listing claim by claim and demotes anything the AI can't find support for in an attached evidence packet. When given accurate evidence labels, it cut the share of unsupported products appearing in the top three results from roughly 65% to 43% — without accidentally burying legitimate listings. But the study's caution matters too: every automated fact-checker they tested failed their reliability bar against human-verified answers, and sloppy or incomplete evidence packets caused a lot of good results to be wrongly suppressed.

The takeaway isn't that AI search is broken forever. It's that ranking by how convincing text sounds is a short-term shortcut with real costs. If you use AI to compare products, insurance plans, or health advice, treat confident detail as a prompt to verify, not a reason to trust. And note the results depended heavily on which AI model was used — one showed barely any effect, another showed none at all.