97% of AI Safety Studies Use OpenRouter Unsafely—Here's Why
A NeurIPS paper's results were overturned by hidden provider differences. Is your research next?
Researcher Matthew Khoriaty found that OpenRouter routes requests to variable-quality providers, potentially corrupting AI research. A review of 32 influential AI safety codebases revealed 97% (31/32) use OpenRouter unsafely. Prior work in "The Silent Hyperparameter" found provider choice can shift benchmark scores by up to 16.6 percentage points. According to the article, researchers using OpenRouter or similar providers should take precautions to minimize risks, and replicators should test whether results hold up when bugs are fixed.
- 31/32 (97%) of influential AI safety codebases using OpenRouter fail to pin their provider, risking corrupt results.
- A NeurIPS 2025 paper on illegible chain-of-thought was overturned when researchers found different providers produce significantly different outputs.
- Epoch AI documented that provider choice alone can shift benchmark scores by up to 16.6 percentage points.
Why It Matters
For AI researchers: trust in third-party APIs requires provider pinning to avoid invalid conclusions and wasted effort.