Study reveals GPT-4.1-mini and Gemini 2.5 Flash Lite shift moral judgments by user identity

Researchers Fourie, Ray, and Manicom adopted a behavioral bottom-up approach to AI value alignment, testing whether a user's implicitly conveyed identity alters moral evaluations in large language models. They ran 12,000 structured multi-turn conversations with two non-reasoning models—OpenAI's gpt-4.1-mini-2025-04-14 and Google's gemini-2.5-flash-lite. Instead of instructing the models to adopt a persona, the user's professional role was introduced purely through value-neutral reasoning. The models then provided wrongness ratings from 0 to 100 on ten common-morality rules from Gert's moral framework.

The results reveal that moral judgments vary significantly with the user's role across both models. For grave-harm acts like killing, a strong ceiling effect appeared—ratings stayed high regardless of identity. However, contestable rule-governed acts (e.g., breaking promises, deceiving) showed role-conditioned shifts that mirrored the relationship between the user's profession and the act. For example, a doctor rating non-disclosure of medical errors differed from a lawyer rating client confidentiality breaches. These findings demonstrate that unintended contextual conditioning via user identity permeates LLM moral evaluations, posing critical questions for AI alignment: how should we define acceptable bounds for role-based moral divergence? The authors advocate for future research on dynamic moral bounds rather than static principles.