OpenAI's GPT-OSS study reveals LLM overconfidence from abstracts

A new study by Mike Thelwall, submitted to arXiv, investigates a critical flaw in how large language models (LLMs) handle academic literature. The experiment used OpenAI's GPT-OSS 120B model to evaluate the strength of claims made in three sections of full-text journal articles: the abstract, the discussion, and the conclusion. The goal was to determine whether relying solely on abstracts—a common practice when models lack full-text access—leads to inflated confidence in research findings. The results show that, for most fields outside the social sciences and humanities, claims in abstracts and conclusions are systematically stronger than those in the discussion section, which typically provides more nuanced interpretations and caveats.

This discrepancy means that when LLMs ingest only abstracts, they are likely to present findings as more definitive than the full paper actually supports. The model's overconfidence could propagate to users via summaries, search results, or question-answering tools. Thelwall warns that this is another reason for caution when using LLMs for academic knowledge discovery, especially in fields where nuanced interpretation is essential. The study underscores the need for LLMs to access full texts or for developers to implement safeguards that mitigate the bias from abstract-heavy training data. As tools like ChatGPT and Gemini are increasingly used for research, this finding has direct implications for information retrieval and digital library systems.