Cloudflare to block AI crawlers from ad pages by default in 2026

Cloudflare is shaking up the AI data economy with a bold policy change. Starting September 15, 2026, the company’s default settings will block any web crawler that mixes traditional search with AI training or agentic use from accessing pages that display ads. The change applies to all new customers, new sites from existing customers, and all existing free users — unless site owners manually opt in. Cloudflare CEO Matthew Prince cited that “the majority of traffic on the Internet is non-human” and that bots surpassed human traffic earlier than expected, necessitating faster action.

Cloudflare specifically called out Google, whose Googlebot crawls for AI Overviews and AI Mode alongside traditional search, giving it “2x more information” than other AI companies. While Google offers the “Google Extended” bot for opt-out, Cloudflare argues this still conflates purposes. To operationalize fairness, Cloudflare is evolving its “Pay Per Crawl” model into “Pay Per Use,” partnering with Ceramic.ai and You.com so publishers get paid when their content generates value — not just when fetched. The company also noted that over 50% of AI crawler traffic is spent re-fetching unchanged pages, wasting bandwidth.