Cloudflare’s new toggle separates model-training opt-outs from search crawling — with a key caveat

Cloudflare has rolled out a Disallow AI Training option that lets site owners signal a no-training preference in robots.txt while still allowing major search crawlers to index pages for search. That distinction matters: many publishers want to stop their content entering large model training sets without sacrificing search traffic, but a misconfigured Cloudflare setting can now block search entirely.

What changed and why it matters

Disallow AI Training is part of Cloudflare’s Training control, one of three related controls alongside Search and Agent. The new setting maps to the extended robots.txt tokens used by Google and Apple, so sites can opt out of model training while preserving search crawling by those operators when the crawlers meet Cloudflare’s trust requirements.

Cloudflare also tightened the behavior of its Block and Block on pages with ads options: those now apply to mixed-use crawlers (bots that perform both search indexing and model training). Selecting Block will fully prevent mixed-use crawlers — including Googlebot, Applebot and Bingbot — from crawling the site. In short, choosing Block is a full block for those operators, not a training-only block.

Who keeps search access: the Accountable designation

Cloudflare created an “Accountable” label for mixed-use crawler operators after discussions with crawler operators that began in July. To be Accountable, an operator must meet or commit to four requirements:

  • Provide an opt-out mechanism for AI training via robots.txt or a similar standard.
  • Offer a way to opt out of AI summaries now and through an integration with Cloudflare next year.
  • Give URL-level visibility into which pages were used for training, plus metrics about how content appeared in search.
  • Assure that opting out of training won’t affect traditional search results.

Cloudflare says Google, Apple and Microsoft meet those requirements. It also lists crawlers from Amazon, Anthropic, Meta and OpenAI as Accountable where those companies run separate search and training crawlers — meaning their search-oriented crawlers can continue to access sites labeled Disallow AI Training while their training crawlers are blocked.

How major vendors interpret the new signal

  • Google: Cloudflare’s Disallow AI Training maps to Google’s Google-Extended robots.txt token, which Google says does not affect a page’s inclusion in Search or its ranking. Separately, Google Search Console controls whether content appears in AI Overviews, AI Mode and Discover’s generative features; that Search Console setting does not control training.
  • Apple: Cloudflare signals an opt-out through Applebot-Extended. Apple documents that Applebot-Extended is not considered in ranking and does not crawl for search; preventing content from appearing in Siri and Search AI answers relies on the nosnippet meta tag.
  • Microsoft / Bing: Cloudflare’s Disallow AI Training does not yet send a no-training preference to Bing because Microsoft hasn’t added support for that robots.txt token. Bing’s current opt-out mechanism is the NOARCHIVE meta tag, which Microsoft says prevents content from being used to train its generative models and from being linked in Copilot and Chat.

Migration, deprecation and timing

Cloudflare will migrate sites that previously used the older Block AI Bots toggle into the new schema. The older Block AI Bots toggle and the Managed Robots.txt feature are set to be deprecated. Under migration, sites that previously blocked AI bots will be assigned a combination of Allow for Search, Disallow AI Training for Training, and Block on pages with ads for Agent, unless the owner changes the setting.

Cloudflare says Google plans to roll out URL-level transparency tools for Google-Extended in the coming weeks and that Apple is working on a similar URL-level tool for next year. Microsoft’s support for a robots.txt no-training preference is targeted for early 2027. Cloudflare also aims to add a single control for how much content is included in AI summaries by early next year.

Practical steps for site owners

Review how your Cloudflare settings will migrate. If you want to prevent AI training while keeping search traffic, use Disallow AI Training rather than Block. Choose Block only if you intend to deny mixed-use crawlers all access — including search indexing.

Also verify vendor-specific controls: use Search Console to manage appearance in Google’s AI features, apply nosnippet where you want to avoid Apple’s AI answers, and continue using NOARCHIVE if you want to keep content out of Microsoft’s training sets until Bing supports a robots.txt token.

What to watch next

The practical effectiveness of Disallow AI Training depends on crawler operators completing their implementations and on Cloudflare’s migration choices. Watch for Google’s URL-level transparency rollout, Apple’s tooling next year and Microsoft’s planned robots.txt support for early 2027. Cloudflare’s upcoming single control for AI summaries will also affect how much of your site can surface in operator-generated summaries.

For publishers and site operators, the immediate task is an audit: confirm which sites should keep search access, which should opt out of training, and whether any current settings will unintentionally block search crawlers.