Google kills free search for AI; Cloudflare, GoDaddy block scrapers

Google is effectively closing its free search index for all but the most limited use cases. Starting January 1, 2027, the company will restrict site-specific searches to just 50 domains per query—a drastic reduction from the previous unlimited tier—and has not yet published any pricing for advanced or unlimited access. This change, combined with a gradual tightening of API access observed over recent months, has already started producing 400 errors from many sites when AI tools attempt to pull web data. The move is widely seen as Google reinforcing its moat, pushing developers and researchers toward paid Google Cloud services or proprietary datasets.

Meanwhile, Cloudflare has made its AI bot challenge the default setting for all customer sites, and a recent partnership with GoDaddy extends that protection to millions of domains. This creates a two-front blockade: search engine data becomes expensive and restricted, while scraping tools face aggressive traffic filters at the edge. For the open-source AI community—especially those relying on local models with internet retrieval (RAG, agents)—the existing infrastructure is crumbling. The immediate effect is a sharp decline in the quality and breadth of web-based knowledge retrieval. The Reddit post calls for an “open project” to fill this gap, likely the next major dependency for AI development. Options include federated search indexes, peer-to-peer crawling networks, or partnerships with smaller search engines like Brave or DuckDuckGo.