Study: LLM Reliability Varies by Question Type and Prompt Wording
One prompt change can flip an LLM's 'belief' while objective answers stay stable.
A new arXiv study evaluated four instruction-tuned model families on three objective datasets (MMLU, ARC, CulturalBench) and three subjective datasets (Political Compass Test, ValueBench, World Values Survey). By applying multiple types of prompt changes—variations in wording, framing, and format—and measuring answer consistency, the researchers found significant effects of model, dataset, prompt category, and their interactions. The dataset type effect was significant, and the interaction between dataset type and prompt category was large. These results indicate that prompt robustness depends on the question type, the prompt change, and the model.
- Tested 4 instruction-tuned LLM families on 3 objective and 3 subjective datasets with multiple prompt variations.
- Objective questions (MMLU, ARC, CulturalBench) showed high robustness; subjective questions (Political Compass, ValueBench, WVS) were highly sensitive to wording changes.
- Significant interaction between dataset type and prompt category (p < 0.05) indicates prompt robustness is task-dependent, not model-wide.
Why It Matters
LLM 'values' from surveys are unreliable; prompt phrasing can flip responses, requiring new evaluation standards.