Why Avoid Generative Legal AI Systems? Hallucination, Overreliance, and their Impact on Explainability
A new paper argues AI systems for law are built for fluency, not facts, creating dangerous confabulations.
A new research paper from Gizem Gültekin Várkonyi, published on arXiv, delivers a stark warning against the rapid adoption of Generative Legal AI (GLAI) systems. The paper defines GLAI as an umbrella term for AI models—like those from OpenAI or Anthropic—specifically adapted for legal tasks, from document drafting to criminal justice decision support. The core argument is that these systems are fundamentally misaligned with legal needs because their underlying architecture is designed for statistical token prediction, not rigorous legal reasoning. This mismatch leads to 'confabulations' or hallucinations, where the AI prioritizes generating fluent, persuasive text over factually accurate or legally sound content.
These hallucinations create a dual risk. First, they obscure the AI's reasoning process, making it impossible for legal professionals to understand how a conclusion was reached. This directly undermines the principle of explainability, a cornerstone of European AI governance frameworks like the EU AI Act. Second, the very fluency and human-like quality of the output can induce automation bias and overreliance, where lawyers or judges may uncritically accept the AI's suggestions. The paper concludes that without robust mechanisms for meaningful human scrutiny, the routine use of GLAI poses a significant threat to judicial independence and the protection of fundamental rights, advocating for caution over convenience.
- GLAI systems are built on statistical models (like GPT-4) for fluency, not legal accuracy, leading to dangerous confabulations.
- The persuasive, human-like output fosters professional overreliance and automation bias, compromising critical scrutiny.
- This dynamic weakens explainability, a key EU AI governance requirement, and threatens judicial independence.
Why It Matters
For legal tech, it's a critical framework for evaluating AI tools, emphasizing human oversight over automation.