Prompt formatting can swing LLM leaderboard results by 30x

A new preprint from researcher Deep Mehta exposes a hidden vulnerability in LLM benchmarking: the way you format a prompt wrapper can change a model's score by up to 30 times. The study proposes two complementary metrics — the Format Sensitivity Index (FSI), which captures accuracy variance across wrappers, and the Parseability Sensitivity Index (PSI), which measures how often the model produces parsable answers. Using 140,000 generations across 7 QA benchmarks, 5 wrapper families, and 4 instruct models ranging from 7B to 72B parameters, Mehta found that FSI differences are largely driven by compliance failures — models that fail to follow formatting instructions produce both lower accuracy and more volatile scores.

A fixed-effects regression confirmed that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper type. The work argues that reporting accuracy without reporting wrapper variance is statistically fragile and can lead to false conclusions in leaderboard comparisons. For practitioners, the paper offers practical recommendations: benchmarking pipelines should report FSI and PSI alongside accuracy, and structured-output deployments should test multiple wrapper variants before selecting a prompt. The findings have direct implications for any team relying on LLM evaluations to make model selection or safety decisions.