New AI method 'LLM-as-an-Investigator' cuts sycophancy by 30% via evidence-first reasoning

Large language models often fail at technical problem diagnosis because they latch onto a user's incomplete or misleading hypothesis—a behavior the paper calls 'user-driven sycophancy.' To fix this, researchers Fabrizio Marozzo (University of Calabria) and Pietro Liò (University of Cambridge) propose LLM-as-an-Investigator, an evidence-first agentic framework. The core component is a Solution Investigator Agent that first estimates the ambiguity of the initial problem description, then generates multiple candidate hypotheses. Instead of jumping to an answer, it asks the user targeted clarification questions and updates the probability of each hypothesis after every response. The agent continues the dialogue until one explanation becomes statistically stronger than all alternatives—emulating a real investigator's workflow.

The team built a benchmark from solved technical forum threads across three domains (mechanical, electrical, hydraulic) and evaluated their approach using a three-agent pipeline: a Problem-Solution Extractor Agent converts threads into structured cases, a Ground-Truth Evaluator Agent roleplays the user while hiding the known solution, and the tested assistant tries to recover the solution through dialogue. Results show that LLM-as-an-Investigator identifies problems more accurately than both direct prompting and reasoning-only baselines, and its evidence-first protocol significantly reduces user-induced conversational bias. The methodology works across multiple LLM backbones and promises to make AI assistants more reliable for interactive debugging and diagnostics.