New arXiv model: Generative search drives web corpus to extinction

Generative search engines (GSEs) like AI-powered answer engines pull content from the web and summarize it directly, capturing value without sending traffic back to the source. This extraction diverts the ad revenue and visits that fund content creation. In a new arXiv paper (2608.15896), computer scientist Sylvain Peyronnet formalizes this dynamic as a tragedy-of-the-commons problem. He treats the crawlable web as a "common-pool resource" defined by three metrics: volume (how much content is available), average quality (the value of that content), and lifetime (how long content remains fresh and relevant). Using game theory, he proves that under publisher responses—such as blocking crawlers or reducing updates—extraction degrades all three simultaneously. Publishers opt out, renewal loses funding, and content becomes more perishable, driving the system toward an erosion threshold beyond which the corpus goes extinct.

Peyronnet's model distinguishes between myopic GSEs, which maximize short-term extraction and cross the threshold, and long-run-oriented engines that stay below it to preserve the commons. Extending the model to multiple competing engines, he proves that symmetric equilibrium extraction rates increase with the number of competitors and converge toward the threshold—meaning competition accelerates erosion. Even under the most favorable assumption for extraction—users who strictly prefer direct answers—the socially optimal extraction rate remains below the erosion threshold and no higher than a single engine's sustainable optimum. The paper closes with seven survival mechanisms for publishers, ranging from selective blocking to licensing and syndication strategies. For the AI industry, this is a warning that the raw material for generative search is finite and at risk if extraction isn't priced or regulated as a shared resource.