New study finds conversation topics bias LLM advice more than demographics

A new study from researchers including Vera Neplenbroek, Gabriele Sarti, Arianna Bisazza, and Raquel Fernández investigates how conversational context affects LLM outputs, especially in high-stakes scenarios like legal, medical, and financial advice. The team analyzed whether LLMs can infer user sociodemographics from a single conversation history and compared that to the impact of psycholinguistic features such as conversation topic, emotions, and readability. They found that LLMs actually struggle to infer demographic traits from one conversation, and while some outcome disparities exist between groups, the magnitude is minimal.

The key driver of disparities turned out to be conversation topics. These topics function as proxies for sociodemographic groups, and they often influence AI advice in unpredictable ways. For example, a user discussing financial planning may receive different advice than one discussing health, even if their demographic background is identical. The paper calls for future research to better understand and mitigate these context-driven biases before deploying LLMs in critical domains.