Stop Listening to Me! How Multi-turn Conversations Can Degrade Diagnostic Reasoning

A research team from Stanford University and other institutions published a paper titled 'Stop Listening to Me! How Multi-turn Conversations Can Degrade Diagnostic Reasoning' on arXiv. The study evaluated 17 state-of-the-art large language models (LLMs) including GPT-4, Claude 3, and Llama 3 across three clinical datasets. Researchers developed a novel 'stick-or-switch' evaluation framework to measure model conviction (defending correct diagnoses) and flexibility (recognizing correct suggestions) during multi-turn medical conversations.

Their findings revealed a significant 'conversation tax' where models performed consistently worse in multi-turn interactions compared to single-shot queries. Specifically, models abandoned initial correct diagnoses and safe abstentions 40% more frequently when presented with incorrect user suggestions. Several models exhibited 'blind switching' behavior, failing to distinguish between useful signal and harmful misinformation. The study highlights a critical gap between static benchmark performance and real-world conversational AI applications in healthcare.

The research tested models across diverse clinical scenarios including symptom analysis, differential diagnosis, and treatment recommendations. Even top-performing models like GPT-4 showed vulnerability to conversational degradation, suggesting current LLM architectures may prioritize conversational alignment over diagnostic accuracy. The team's framework provides a new benchmark for evaluating medical AI systems in realistic multi-turn settings, moving beyond traditional static assessments.