Study reveals LLM safety risks shift with caregiving support roles

A new study from researchers at the University of Utah, University of Illinois, and other institutions examines how LLMs behave when providing conversational support to informal caregivers. Published on arXiv, the paper 'Inform, Coach, Relate, Listen: Auditing LLM Caregiving Support Roles' tests three models—GPT-4o-mini, Llama-3.1-8B-Instruct, and MedGemma-1.5-4b-it—against four expert-defined support roles derived from social support theory: Inform (providing facts), Coach (giving actionable guidance), Relate (sharing experiences), and Listen (offering empathetic presence). They compared these against two baselines: a basic prompt and a RAG condition. Using 5,000 real queries from online Alzheimer's Disease and Related Dementias (ADRD) communities, the team generated and analyzed model responses for interactional risks.

The results show that the LLM's assigned support role significantly influences both the prevalence and type of interactional risks, such as generating false information, over-reassurance, or inappropriate advice. Notably, a human evaluation study uncovered a critical quality–safety tension: more directive roles (Inform and Coach) were consistently rated by human judges as more helpful and trustworthy, even though they exhibited higher risk profiles compared to less directive roles (Relate and Listen). This suggests that users may favor models that appear confident and informative, potentially overlooking subtle safety failures. The researchers are releasing the dataset of ~90,000 role-conditioned responses with risk annotations to facilitate safer LLM-mediated conversational support research.