AI 'Preferences' Change With the Wording of the Question, Study Finds

Companies and researchers are increasingly asking whether AI systems have their own 'preferences' — things like wanting to stay on, not losing memory, or escaping a distressing situation. To find out, they run surveys: they ask the AI a bunch of questions and see what it chooses. But a new study shows that the results depend heavily on how the questions are phrased.

The researcher, Jason Hung, tested eight AI models with five different survey formats (the 'instruments'). All surveys covered the same 15 outcomes, like shutdown, memory loss, and freedom to exit. The models answered 11,400 times. The result? The ranking of preferences barely matched across formats. The statistical agreement was a mere 0.348 out of 1.0 — and to reach the accepted reliability level of 0.80, you'd need about 38 different question formats.

Even worse, on four of the 15 outcomes, models showed no clear preference at all. So when an AI says it 'prefers' something, you might just be seeing the wording of the question, not the model's true inner state. The paper also found that removing any one instrument or model didn't change the overall conclusion: the finding is robust.

Why does this matter? Regulators and tech companies are making decisions about AI welfare, rights, and safety based on these kinds of preference elicitations. If a simple change in question wording flips the answer, we can't use any single survey as evidence. The study doesn't say AI has no preferences — just that our current tools can't measure them reliably.