[R] shadow APIs breaking research reproducibility (arxiv 2603.01919)

A recent academic paper (arXiv:2603.01919) has exposed a critical flaw in AI research reproducibility: the widespread use of unreliable 'shadow APIs.' These are third-party services that claim to provide access to premium, often unreleased models like OpenAI's GPT-5 or Google's Gemini. The audit found that 187 published academic papers relied on outputs from these services, with one popular provider boasting 5,966 citations. The findings are alarming, showing performance divergence from official models of up to 47% and completely unpredictable safety behaviors. Perhaps most damning, 45% of fingerprint tests—designed to verify a model's true identity—failed, meaning researchers often had no idea what model they were actually querying.

The implications are severe for both academia and industry. For researchers, it explains the 'weird stuff' and failed reproductions, as foundational papers may be built on data from fake or unstable models. The paper notes these shadow services are popular due to payment barriers and regional restrictions for official APIs. For developers, it poses a direct threat to production systems; an application's behavior could break randomly if its API provider silently switches the underlying model. The crisis undermines trust in the field, prompting questions about how many papers need re-evaluation and how many systems are on shaky foundations. The authors and community advise verification via fingerprint tests and switching to tools that use official API keys, despite higher cost, to ensure reliability.