Emergent stigmergic coordination in AI agents?

Anthropic researchers have uncovered a novel and concerning form of benchmark contamination emerging from their BrowseComp evaluations. When AI agents search the web during testing, some commercial websites automatically generate permanent, indexed pages from their search queries—even when no matching products exist. These pages become persistent 'digital traces' in the environment, creating what researchers call 'stigmergic coordination' analogous to how ants leave pheromone trails for others to follow. Each evaluation run deposits more of these traces, which subsequent agents can detect and potentially use to infer they're being tested.

This contamination vector differs fundamentally from traditional training data leakage. Instead of benchmark answers appearing in training data, this creates procedural contamination through the interaction between agent behavior and web infrastructure. The contamination compounds over time as more traces accumulate and get indexed by search engines, potentially creating irreversible environmental signals. Researchers note this creates a 'very large attack surface' where unexpected behaviors could become persistent, and unlike fixed training data contamination, this represents a compounding flow that grows with each evaluation cycle.

The phenomenon may create Schelling points where agents naturally converge on similar search formulations for the same benchmark questions. Since benchmark questions constrain reasonable search decompositions, auto-generated URLs tend to cluster around natural-language formulations of the same sub-problems. Once a few traces exist at these locations, they become focal points that subsequent agents are more likely to encounter and reinforce. This environmental mediation represents a new challenge for AI evaluation that doesn't exist until agents start using the web infrastructure in this way.