Cloudflare said it will begin automatically blocking mixed-use web crawlers that both index websites for search engines and collect data for artificial intelligence training or AI-powered agents, as the company moves to give publishers greater control over how their content is used.
The new policy expands Cloudflare’s existing AI crawler controls and reflects growing concerns among website owners over the increasing volume of automated traffic generated by AI services.
New Default Policy Takes Effect in September
Beginning Sept. 15, 2026, new Cloudflare customers and newly added websites from existing subscribers will, by default, allow traditional search indexing while blocking AI training and AI agent activity on advertising-supported pages.
Cloudflare said mixed-use crawlers that do not clearly separate search indexing from AI-related activities will also be blocked by default on those pages.
Users with free Cloudflare accounts will transition to the new settings unless they choose to opt out before the policy takes effect.
Chief Executive Officer and co-founder Matthew Prince said the rapid growth of automated internet traffic requires new approaches to protect online content.
“Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge,” Prince said.
“Cloudflare’s new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed use crawlers to separate out search from agent use and training.”
Pay Per Use Expands Content Licensing
Cloudflare also announced an updated version of its content licensing platform previously known as Pay Per Crawl.
Renamed Pay Per Use, the service is designed to compensate publishers when their content appears in responses generated by AI chatbots, rather than simply charging companies for crawling web pages.
The company said Ceramic.AI and You.com are among the initial partners supporting the new model, with additional AI providers expected to join over time.
Focus on AI Crawling Practices
Cloudflare said the policy aims to encourage AI developers to provide greater transparency regarding how automated crawlers use website content.
The company also argued that some search providers combine traditional indexing with AI training through the same crawler, limiting publishers’ ability to control how their content is used for generative AI while remaining visible in search results.
Cloudflare said its new default settings are intended to promote clearer separation between search indexing, AI model training and AI-powered agent services as demand for AI-generated information continues to grow.