New study finds LLMs prioritize lexical overlap over semantic meaning
Lexical bias persists across all architectures and training regimes, even in semantic models.
A team of researchers from multiple institutions (Rizwan, Haider, Subramani, Diab, Siddique, Sajjad) has published a study on arXiv revealing that large language models (LLMs) consistently rely more on lexical overlap—matching surface-level word patterns—than on genuine semantic content when forming internal representations. Using adversarial semantic stress tests and information-theoretic analysis, they tested models across various architectures (including LLaMA, GPT variants), training regimes (pretrained vs. fine-tuned), and objective functions, even models explicitly trained for semantic similarity. In every case, lexical influence dominated, indicating a fundamental structural bias in how LLMs encode meaning.
The authors also discovered a striking 'mid-depth' region in the models where both lexical and semantic signals simultaneously degrade, suggesting a transitional zone where representations are poor for both surface form and deeper meaning. They demonstrated the practical impact of this lexical bias through case studies on summarization (where it causes hallucinated or irrelevant content) and model editing (where it hinders accurate updates). This work highlights a critical limitation in current LLMs: they may appear to understand language but often rely on word-matching shortcuts, raising concerns for any application that demands robust comprehension.
- Lexical bias persists across all tested architectures (LLaMA, GPT) and training objectives, including models fine-tuned for semantic similarity.
- A mid-depth layer region exists where both lexical and semantic signals degrade simultaneously, creating a representation 'dead zone.'
- Downstream tasks like summarization and model editing are measurably harmed—e.g., summaries include irrelevant words due to lexical matching.
Why It Matters
Reveals a core weakness in LLM comprehension, affecting reliability of summarization, editing, and other semantic tasks.