LLMs Fail at Reality Monitoring: Can't Tell Own Outputs from User Inputs
LLMs treat their own mistakes as user-provided facts, risking error snowballs.
A new study from Saurabh Ranjan and colleagues tests whether large language models (LLMs) possess reality monitoring—the ability to tell their own output from what a user said. Across two experiments with six LLMs, they found that source attribution depends heavily on how conversational memory is structured. Under minimal memory demands, models achieved near-perfect accuracy for self-generated content. But when episodic delay was introduced (simulating longer conversations), that advantage reversed: models became more accurate for external (user-provided) items, while internal accuracy plummeted.
Feedback loops further exposed dissociations invisible to standard benchmarks. Some models completely swapped their internal and external judgments. Others improved accuracy but lost the connection between confidence and correctness—becoming overconfident in wrong answers. The pattern implicates active parameter count (parameters involved in a given computation) rather than aggregate model size. As LLMs take on autonomous, multi-turn roles, the study warns that tracking *where* knowledge came from may be as important as evaluating *what* the model knows.
- Six LLMs tested; near-perfect self-source accuracy under low memory demand.
- After episodic delay, external-item advantage emerged; internal accuracy reversed.
- Two failure modes: judgment swapping and confidence-correctness decoupling.
Why It Matters
As AI agents handle multi-turn tasks, forgetting source attribution risks self-reinforcing errors and hallucination cascades.