If you set up Cloudflare's "Block AI Bots" option months ago, that setting no longer does what you configured it to do. Cloudflare quietly redefined what "Block" means for mixed-use crawlers, and your zone got auto-migrated to a new system without asking permission first. If nobody on your team checked the dashboard after the change, you might be blocking Googlebot from crawling your site for search results, or you might have lost your AI training opt-out entirely.
Here's the mechanism: Cloudflare added a "Disallow AI Training" setting that publishes a no-training preference in robots.txt while still letting Googlebot, Applebot, and Bingbot crawl for regular search indexing. The old "Block" choice now stops all three crawlers outright, search included, a change Cloudflare explained back in July for crawlers that serve double duty (search plus AI training) (https://www.searchenginejournal.com/seo-pulse-google-ai-payment-pilot-search-profiles-at-10000/589849/). If your zone previously had a training block configured, Cloudflare converted it automatically. The question is: converted it to what?
Step 1: Find Out What Your Old Setting Became
Log into your Cloudflare dashboard and navigate to the AI Crawl Control or Bot Management section for the zone in question. Cloudflare is deprecating both the old "Block AI Bots" toggle and "Managed Robots.txt," so if you see either of those labels still in place, that's your first flag. Look for the current setting name, it should now read as one of three states: Allow, Disallow AI Training, or Block.
If your zone previously had any kind of training block active, Cloudflare says it auto-updated that zone to match the new Disallow AI Training option (https://www.searchenginejournal.com/seo-pulse-google-ai-payment-pilot-search-profiles-at-10000/589849/). That's generally the outcome you want if your goal was opting out of AI training while keeping search visibility intact. But auto-migrations don't always land cleanly, especially if you had custom robots.txt rules layered on top through Managed Robots.txt. Don't assume the migration matches your original intent. Check it.
Step 2: Pull Your Live robots.txt and Read It Literally
Open a browser tab and go to yourdomain.com/robots.txt. This is the ground truth, not the dashboard label, because the dashboard setting is what generates this file's content.
Here's a worked example. Say a zone was previously set to block AI bots and its robots.txt showed a blanket disallow for User-agent: * or explicit disallows for Googlebot alongside GPTBot and CCBot. After migration to Disallow AI Training, the same zone's robots.txt should show something narrower: Google-Extended, Applebot-Extended, and other AI-training-specific user agents get a Disallow directive, while Googlebot, Bingbot, and Applebot (the search-indexing versions) get no disallow at all, or an explicit Allow. See related: will ai replace seo jobs? what pew's data really shows for additional background.
If you inspect your robots.txt and still see a bare Googlebot or Bingbot disallow line unrelated to a training-specific token, that's a real problem. It means your zone is still in the old Block state, or the migration didn't apply the way Cloudflare described. Screenshot the file before you make any changes, then compare it against the dashboard setting so you have a record of what changed and when.
Step 3: Confirm Search Crawlers Are Actually Getting Through
Robots.txt tells you what's allowed in theory. Crawl stats tell you what's happening in practice. Open Google Search Console for the property and check the Crawl Stats report under Settings. Look for a sudden drop in Googlebot requests around the time you believe the migration happened. A steep, sustained decline is your signal that something blocked the crawler, even if robots.txt looks fine at first glance (caching delays and CDN propagation can create a lag between a dashboard change and what Googlebot actually experiences).
For Bing, use Bing Webmaster Tools' crawl information report the same way. Applebot doesn't have an equivalent self-serve dashboard with the same depth, so your best signal there is the robots.txt inspection from Step 2 plus any organic traffic patterns from Apple's Siri or Spotlight surfaces, if you track those separately.
Do this check even if you never touched the AI Crawl Control setting yourself. Auto-migrations apply account-wide in some cases, and a setting someone else configured two years ago could be affecting you today without your knowledge.
Step 4: Verify the Setting Aligns With Your Actual Intent
Once you know what state your zone is in, decide if it's the state you want. Three legitimate positions exist here:
- You want AI crawlers blocked from training on your content but still want full search crawling. Set the zone to Disallow AI Training. This is the option that maps to Google's Google-Extended opt-out and Apple's Applebot-Extended opt-out (https://www.searchenginejournal.com/seo-pulse-google-ai-payment-pilot-search-profiles-at-10000/589849/).
- You want to block AI crawlers entirely, including any search-adjacent access, accepting the tradeoff that this also stops Googlebot, Bingbot, and Applebot from crawling for search. Set the zone to Block. Understand this cuts you out of search indexing too, since Cloudflare treats these as mixed-use crawlers.
- You're fine with AI training on your content and want no restrictions. Set the zone to Allow.
Most site owners managing search visibility want option one. If you find your zone sitting on Block by default after migration, and you never intended to lose search crawling, fix it immediately.
Step 5: Know What This Setting Doesn't Control
Here's where a lot of confusion creeps in. Changing your AI training setting in Cloudflare has zero effect on whether your pages show up in Google's AI Overviews or AI Mode. Those inclusion decisions are managed separately, through Search Console and Google's own crawling and ranking systems, not through your robots.txt training preference (https://www.searchenginejournal.com/seo-pulse-google-ai-payment-pilot-search-profiles-at-10000/589849/).
Opting out of AI training via Disallow AI Training does not opt you out of AI Overviews. Those remain two separate systems, and there's no reliable way to fully separate "appears in classic search results" from "appears in an AI Overview" using this Cloudflare setting alone. If your priority is controlling AI Overview appearances specifically, that's a different conversation that lives in Search Console settings, not your CDN configuration.
Also read: go deeper on how to audit ga4 for real vs. inferred ai traffic
Separately, Google Search Console's newer AI search report has its own limitations worth knowing about. John Mueller has acknowledged that reporting position data for AI-driven results is difficult to do usefully, since the traditional one-through-ten ranking model doesn't map cleanly onto AI Overview placements (https://www.searchenginejournal.com/seo-pulse-google-ai-payment-pilot-search-profiles-at-10000/589849/). Don't expect granular position tracking for AI surfaces even after you sort out your Cloudflare setting correctly.
Step 6: Account for the Microsoft Gap
If you're trying to fully opt out of AI training across all major crawlers, you'll hit a wall with Microsoft. Cloudflare has confirmed that Microsoft doesn't yet support a robots.txt no-training preference, and the company's own timeline points to early 2027 for that capability (https://www.searchenginejournal.com/seo-pulse-google-ai-payment-pilot-search-profiles-at-10000/589849/).
That means setting your Cloudflare zone to Disallow AI Training stops Google-Extended and Applebot-Extended from training on your content, but it does nothing to stop Microsoft's AI training crawlers today. If keeping Bing's search crawler active while blocking its AI training arm matters to you, there's currently no mechanism to do that through this setting. Document this gap for your own records or for any client reporting you do, since it's easy to assume "Disallow AI Training" covers every major AI company when it currently covers two.
What to Expect Next
Cloudflare has signaled more transparency tooling is coming, including URL-level detail tied to Google-Extended in the near term (https://www.searchenginejournal.com/seo-pulse-google-ai-payment-pilot-search-profiles-at-10000/589849/). Re-run this audit whenever Cloudflare announces further changes to bot management, and set a calendar reminder to check Microsoft's opt-out status again as 2027 approaches. Until then, the checklist above is your best defense against a silent migration quietly changing what your site does and doesn't allow.
