Fine-tuned Llama-2 hallucinates more on domain-specific queries, study finds

A new study by Sanchita Porwal, Sai Prasath S, Xingjian Bi, and Madelyn Scandlen, published on arXiv, evaluates hallucinations in domain-adapted LLMs. The researchers fine-tuned Meta's Llama-2 model using the Lamini dataset—a collection of instruction-following examples with a focus on domain-specific knowledge. They designed experiments to test three capabilities: memorization (recalling exact training examples), recall (retrieving relevant facts from the training data), and reasoning (applying knowledge to novel scenarios). The goal was to see how well fine-tuning alone can help a general-purpose model adapt to a specialized field without introducing errors.

The results reveal significant limitations. While the model showed proficiency on queries similar to its training data, it struggled to accurately reason about new domain-specific information, leading to frequent hallucinations. Notably, the model displayed a tendency to “over-generate”—providing correct answers but appending extra, often unfaithful, details. This suggests that fine-tuning alone does not teach the model to distinguish between known and unknown information, and may even encourage verbosity at the expense of accuracy. The authors highlight that domain adaptation via fine-tuning remains brittle, and more robust methods—such as retrieval-augmented generation or confidence calibration—are needed to safely deploy LLMs in specialized contexts like medicine, law, or finance.