A CDN move can look perfect in a browser and still lock Googlebot out. Google's guidance, as summarized in Search Engine Journal's weekly roundup, names two risks. Blanket bot protection can make Googlebot fail CAPTCHA challenges, and a brand new CDN starts with an empty cache, so early crawl requests all land on your origin server.
The roundup also notes that if challenges persist, pages can start dropping out of search results. You probably don't control the CDN or firewall, so your job is to test, document, and hand the owner a request they can check. This guide gives you the audit for both sides of the switch.
What these checks can and can't tell you
Every check below shows current behavior. They tell you whether verified Googlebot gets real pages right now. They do not predict rankings or traffic, and a clean audit guarantees nothing about search results.
Results also differ by CDN vendor and plan. Some plans include verified-bot lists, some expose challenge data in logs, and some don't. If a setting below doesn't exist on your plan, say so in your request and ask what the equivalent is.
Phase 1: Before you switch traffic
Step 1: Set a baseline from your current logs
Pull a week or two of origin server logs while the old setup is still live. Filter for requests with a Googlebot user agent, then count status codes and note the average response size per URL template. You need this later to spot challenge pages, which are often small and sometimes return a 200.
Step 2: Verify which hits are really Googlebot
A user agent string proves nothing, because anyone can fake it. Google's documented method has two parts. Run a reverse DNS lookup on the requesting IP and confirm the hostname belongs to a Google domain such as googlebot.com or google.com. Then run a forward lookup on that hostname and confirm it returns the same IP. Google also publishes its crawler IP ranges, so check its current crawler documentation for the lists.
Do this for a sample of IPs, not every line. Group hits by IP, verify the top ones, and mark the rest as unverified. Unverified traffic claiming to be Googlebot is exactly what your bot protection should challenge, so keep it out of your Googlebot numbers.
Step 3: Ask for these settings before the cutover
Work through this list with whoever owns the CDN and firewall:
- Verified bots are allowed in the CDN and the firewall or WAF, and the setting is on before bot protection is enabled. Google's advice, per the roundup, is to turn this on first.
- The allowance relies on verification (reverse DNS or the vendor's verified-bot list), not on the user agent string alone.
- No rule serves a CAPTCHA, JavaScript challenge, or interstitial to verified Googlebot on any path, including robots.txt, sitemaps, and CSS and JavaScript files.
- Rate limiting rules exempt verified crawlers, or are set high enough that crawl bursts won't trigger them.
- Challenge and block events are logged with a field you can filter on, and you can get read access to those logs.
- The origin can handle a burst of cache-miss traffic. Ask what the origin's capacity is and who gets alerted if it starts returning 5xx errors.
Step 4: Pre-warm the cache
A cold cache means the first request to every URL goes to your origin. Crawlers hit far more URLs than visitors do, so the origin takes the load. Google's suggestion is to fetch your most popular and most frequently crawled URLs before switching traffic. For more on this, see read about wordpress sitemap showing raw xml in chrome? audit steps.
Build the list from three places: the most-requested URLs in your analytics, the most-crawled URLs in your Googlebot log baseline, and one or two representative URLs per page template. Then request each one through the CDN. Before DNS changes, many CDNs let you test through a staging hostname or by pointing your machine at the edge IP. Check your vendor's docs for how.
A simple check for one URL is curl -s -o /dev/null -D - https://www.example.com/page/. Look at the response headers for a cache status header. The header name and values vary by vendor, so confirm what yours uses. Run it twice. The second response should show a hit.
Pre-warming has limits:
- Many CDNs cache per edge location, so warming from one place may not warm the location Googlebot uses. Whether your plan uses shared or tiered caching is a question for the vendor.
- Cache rules, query strings, cookies, and Vary headers can make "the same URL" cache as separate objects.
- Entries expire. Warm the cache close to the switch, and check your TTLs.
- You can't warm what you don't know crawlers want. Expect some misses on long-tail URLs regardless.
A hypothetical worked example: reading a log for challenged Googlebot
These log lines are invented for illustration. They are not real test data, the IPs are placeholders from documentation ranges, and the numbers mean nothing outside this example. Suppose the fields are IP, time, request, status, response bytes, user agent, and a CDN action field.
198.51.100.20 [12/Dec 03:14:07] "GET /products/blue-widget/" 200 48210 "Googlebot/2.1" action=allow
198.51.100.21 [12/Dec 03:14:09] "GET /products/red-widget/" 200 5120 "Googlebot/2.1" action=challenge
198.51.100.22 [12/Dec 03:14:11] "GET /products/green-widget/" 403 4821 "Googlebot/2.1" action=block
203.0.113.50 [12/Dec 03:14:12] "GET /products/blue-widget/" 403 4821 "Googlebot/2.1" action=block
Line 1 looks healthy: a 200, a page-sized body, and an allow action. Line 2 is the tricky one. The status says 200, but 5,120 bytes is far smaller than the 48,210 bytes for a normal product page, and the action says challenge. Googlebot likely received a challenge page, not your product. A status code report alone would miss this. For more on this, see full coverage of google automated calls business profile setting audit.
Line 3 is a plain block on a request that claims to be Googlebot. Line 4 is the same story from a different IP. Here's the decision step: run reverse and forward DNS on 198.51.100.21, 198.51.100.22 and 203.0.113.50. If the first two resolve to a Google hostname and back, they are real Googlebot being challenged or blocked, and you have a specific finding for the firewall owner. If the third fails verification, the block is correct and you leave it alone.
Phase 2: After you switch traffic
Step 5: Run the URL Inspection live test
In Search Console, inspect a handful of key URLs and run the live test. Check that the page is fetched successfully and that the rendered HTML shows your real content, not a challenge page. Do this for each template and for a few deep URLs. A live test is one fetch from one moment, so a pass doesn't rule out intermittent challenges.
Step 6: Watch the Crawl Stats report
Open Crawl Stats in Search Console settings and compare against your baseline over the following days. Warning signs include a rising share of server errors (5xx), more 403 or 429 responses, slower average response time, or a sharp fall in total requests.
A cold cache hitting your origin can show up as slow responses or 5xx errors. Challenges tend to show up as 403s or a crawl that quietly shrinks. The report is aggregated and delayed, and it can't tell you why a request failed, so treat it as a smoke alarm, not a diagnosis.
Step 7: Re-run the log analysis
Repeat Steps 1 and 2 on CDN or origin logs. Verify IPs again, then compare status codes, response sizes, and the CDN action field against your baseline. Look for verified Googlebot IPs with challenge or block actions, and for 200 responses with unusually small bodies. Also check the cache status field: a hit rate that climbs over the first days is what you want to see.
Also read: also worth reading: search console regex filters for conversational queries
Logs show what happened to requests that reached your infrastructure. They can't show requests that never arrived, and they can't show what Google did with a page afterward.
A request template for the CDN or firewall owner
Keep it specific and checkable. Adapt this:
Before we switch traffic to the CDN, please confirm the following and send us evidence (screenshot or rule export). 1) Verified bots are allowed in both the CDN and firewall, and this is enabled before any bot protection rule goes live. 2) The allowance uses IP or reverse-DNS verification, not user agent alone. 3) Verified Googlebot never receives a CAPTCHA, JavaScript challenge, or block on any path, including /robots.txt and sitemaps. 4) Rate limits exempt verified crawlers. 5) We get read access to logs that include client IP, status, response bytes, cache status, and the security action taken. 6) Please warm the cache for the attached URL list (top traffic, top crawled, one per template) shortly before cutover, and tell us the cache TTL and whether caching is per location. 7) Name the person we contact if crawl errors rise after launch.
What to expect next
The first few days after the switch are when this audit pays off. Schedule the log comparison and the Crawl Stats check for that window instead of waiting for a traffic report.
If verified Googlebot shows up in challenge or block lines, send the owner the log lines and the DNS verification results, and ask which rule fired. Vendors and plans differ, so the fix may not match Google's general guidance exactly. The evidence still gives the owner a specific place to start.



