New AI Research: Same Question, Different Wording, Different Answer

If you ask an AI chatbot the same question two ways — "What's the weather?" versus "Can you tell me the weather?" — you might get different answers. Researchers call this "instruction sensitivity." This paper looked at whether standard training, called supervised fine-tuning (showing AI many examples of right answers), makes that inconsistency go away.

The researcher tested several AI models of different sizes. For smaller models, training helped a lot: their answers became 54–71% more consistent across differently worded questions. But for the largest model tested, the improvement was so small it couldn't be distinguished from random chance. Interestingly, one model showed the pattern while another similar one didn't, meaning the effect depends heavily on the specific AI.

The study also found something sneaky: how you measure consistency changes the conclusion. When AI was asked to write an answer freely, it looked less robust. When it was scored by picking the correct answer from options, it looked more robust — even when overall accuracy was similar. So whether an AI seems "confused" by rephrasing can depend on the test method, not just the model.

For everyday users, this means don't assume an AI will give the same answer if you rephrase something important. For developers, it's a warning that training alone doesn't automatically make AI reliable. Building AI that responds consistently needs better training methods and more thorough testing — with many phrasings and multiple scoring approaches. Until then, if an answer matters, it's worth asking twice in different words.