Why AI’s Citations Might Be Tricking You

AI systems now often include citations at the end of their answers—think of them like footnotes that supposedly prove where the chatbot got its facts. This practice is meant to reduce "hallucinations" (made-up answers) and help users verify claims. But a new study from four researchers at the University of Maryland and Johns Hopkins University tested whether humans and AI models actually find citations helpful when comparing answers to science questions.

The team asked both people and several open-source AI models to pick the better answer between pairs that included citations. They found that humans preferred answers with fewer citations, but those citations were more varied and diverse. In other words, people didn’t want a long list—they wanted a few high-quality sources. But when AI models were asked to choose, their preferences were inconsistent and depended on the specific model and data used. Worse, the AI models couldn’t actually read the cited sources, so their preferences weren’t based on real evidence.

The researchers say this could be a problem because AI’s “preferences” during training might be unreliable, potentially leading to answers that sound convincing but aren’t actually better or more trustworthy. The team is calling for better ways to collect human feedback that truly reflects trustworthy information—so AI learns to value good sources, not just long ones. The findings could influence how companies train AI in the future, especially in areas like health or law where trust and accuracy matter most.