A CDN migration changes the route between your website and its visitors. It also changes the route used by search crawlers. A fast home page after the switch is encouraging, but it does not prove that product pages, older articles and essential resources remain available under sustained traffic.
On 11 November 2026, Google published guidance on CDN migrations and crawling. Faster responses can increase crawling when demand exists, while requests for uncached resources still reach the origin server. A second risk appears when teams respond to load by activating blanket bot challenges: legitimate crawlers can lose access.
For founders, ecommerce teams and technical leads, the useful question is whether the migration preserves reliable delivery across the whole site. This guide turns that question into a practical acceptance plan covering cache behaviour, crawler identity, security rules and recovery.
Understand the two systems you are changing
The origin is the application or server that produces your content. The CDN is an additional delivery layer that can serve eligible cached responses closer to visitors. Keep separate records of what happens at each layer. An edge response may look healthy while the application behind it is processing expensive requests.
Google's crawl-budget documentation separates capacity from demand. A responsive server can support more crawling, but that does not guarantee that Google will request more pages. Crawling also does not guarantee indexing. The advanced budget guidance mainly concerns large or rapidly changing sites; a small service business usually needs a proportionate availability check.
Define the scope of your project before testing. A CDN cutover with unchanged URLs is different from a simultaneous redesign, domain change or catalogue restructuring. Combining those changes makes a later regression harder to diagnose. Where practical, give each release its own evidence and rollback decision.
Confirm what the CDN actually caches
Moving traffic through a CDN does not automatically put every page in its cache. Cloudflare's default-cache documentation, for example, says HTML and JSON are not cached by default, while robots.txt is. Your provider, plan, application and rules determine the behaviour you will actually receive.
Build a small URL matrix before launch. Include a home page, important category, product or service page, older content page, image, script, stylesheet, robots.txt and sitemap. Add account, basket and checkout routes where applicable. For each, record whether caching is intended, whether the response varies by visitor, and who owns the decision.
Inspect repeated GET requests rather than relying on a browser reload alone. Record the status, content type, cache header and returned body. In Cloudflare's response terminology, HIT means the resource was found in cache; MISS means an eligible response was absent and came from the origin. DYNAMIC means the request was not eligible, while BYPASS reflects a response that was ultimately not cacheable.
Those distinctions prevent a common diagnostic mistake: repeatedly requesting a deliberately uncached page will not turn it into a useful cache hit. Conversely, a hit on a personalised response is a reason to investigate isolation, not celebrate a better ratio. Test signed-in and signed-out journeys separately, using safe test accounts.
Review exception rules before increasing cache coverage. Public product information may have different freshness requirements from images, while account details and basket state need correct user separation. Treat a broad cache rule as an application change with functional tests and a documented reversal path.
Establish a baseline the whole team can use
Choose a representative pre-migration period that includes normal trading patterns. For an India-based business with international customers, make the timezone explicit and compare equivalent hours. A quiet local morning may overlap with a busy overseas market. Record campaigns and scheduled imports that could explain unrelated traffic changes.
Keep the baseline focused on decisions:
- Origin health: request volume, response time, application errors and the resource limits your hosting team actually monitors.
- Delivery: edge and origin status codes, cache outcomes and examples of complete responses for important URL groups.
- Crawler access: verified crawler requests, requested paths, challenge events and unsuccessful fetches.
- Customer tasks: a working enquiry, product selection or checkout journey, with expected confirmation behaviour.
A technical SEO audit is useful here when it produces a concrete acceptance set: which URLs and signals must survive the cutover, who checks them, and what constitutes a release-stopping failure. A list of generic recommendations is less useful during an incident.
Assign an incident lead and separate application, security and search owners. Set thresholds from your own capacity and service requirements. There is no universal safe number of crawler requests per second, cache-hit percentage or migration duration that this guidance can supply.
Protect crawler access without weakening security
Separate a request's claimed identity from its verified identity. An HTTP client can supply a Googlebot user-agent string; that label alone is not an access credential. Google documents verification through published IP ranges or reverse and forward DNS checks. Different crawler and fetcher categories use different lists.
Ask your security owner to document how the CDN identifies trusted crawlers and which exact rules they bypass. The desired outcome is access to the public content those crawlers need. It should not become a blanket exemption for every request claiming to be a bot, or permission to enter authenticated areas.
Test a legitimate public URL, a protected account route and a known unwanted request pattern. Save the matching security event or rule identifier where available. This makes it possible to show that a narrow exception fixed access while the relevant protections remained active.
Remember that a successful manual browser visit tests your own session and location. It does not establish what a crawler received elsewhere. Pair public fetch tests with actual verified crawler logs. When those disagree, investigate the delivery route, rule match and response body before changing content or indexing settings.
Run the cutover as a controlled release
Prepare a bounded warming set
Google recommends controlled cache warming for important and frequently crawled URLs. Turn that recommendation into a reviewed list with an owner and a rate limit. Include commercially important routes and the resources they need, then test whether those responses are eligible for caching.
Keep warming within a load level the origin can sustain. It is easy to create the very pressure you are trying to avoid by launching an unrestricted fetch of a large catalogue. Warming a sample is an operational preparation step, not proof that every edge location or URL is ready.
Preserve the content contract
Compare the public response before and after routing changes. Check the main content, canonical destination, indexing directives, redirects and structured data on the acceptance set. Also confirm that robots.txt and the sitemap contain the intended current version. Keep a saved response for any discrepancy so developers can reproduce it.
For a commerce site, run product discovery, variant selection and checkout checks as part of the same release. ecommerce SEO support should connect crawl access with usable catalogue pages: a technically reachable URL is insufficient when prices, availability or customer actions are incorrect.
Watch both sides of the connection
Keep origin and CDN observations together during the rollout. Record the time of each configuration change and avoid changing several security and cache settings at once. If a problem appears, a short sequence of isolated changes is much easier to reverse and explain.
Diagnose the symptom before choosing a remedy
Use the following as a triage sequence, then confirm the cause in your own logs. The patterns are practical examples rather than a prediction that every migration will encounter them.
- Healthy pages, rising origin work: inspect which URL groups are reaching the application and whether their cache policy matches the plan. Check for unexpected query variations or intentionally dynamic routes before expanding capacity or caching.
- Challenge pages or access denials: identify the security rule and verify the affected crawler. Repair the narrow rule responsible, then repeat the failed request and check subsequent crawler access.
- Old content after an update: compare origin and public responses, identify the relevant cache entry or variant, and use the smallest appropriate invalidation. Retest content and headers together.
- Failures concentrated in customer journeys: review session and application behaviour. Pause rollout expansion if a customer cannot complete the critical task, even when the public landing page is fast.
A helpful incident record contains a timestamp, affected URL group, observed response, suspected layer, corrective action and verification result. Keep the original evidence as well as the successful retest. This gives the next shift a usable handover and prevents an unexplained setting from becoming permanent.
Use emergency crawl reduction sparingly
If crawler traffic is causing critical load, Google's emergency crawl-rate guidance documents temporary 500, 503 or 429 responses. Significant errors can reduce crawling across the hostname, including otherwise healthy URLs. With 503 or 429, a Retry-After header can indicate when to retry.
This is an emergency control with consequences: fewer pages may be discovered or refreshed, and repeated errors over multiple days can lead to removal from Google's index. Google's guidance warns against maintaining it beyond a short period, such as a few hours or one to two days.
Before applying an emergency rule, name its owner, record the reason and set a review time. Once capacity recovers, remove the temporary condition and verify ordinary responses. Check that an intermediate cache is not continuing to serve the incident response after the application has recovered.
Do not treat a successful traffic reduction as the incident's only outcome. The release is stable when users can complete their tasks, necessary crawlers can fetch public content and the origin operates within agreed limits. Capture all three in the closure note.
Measure migration quality separately from rankings
For the immediate review, report availability, response correctness, crawler access and customer-task completion. Then follow indexing and organic performance over an appropriate comparison period. Annotate the migration date and other major events rather than assigning every movement to the CDN.
This distinction is especially relevant now. Google's November 2026 core-update notice records a global rollout beginning on 10 November and says it may take up to two weeks. A migration during that period needs an explicit measurement caveat; a ranking movement alone cannot isolate its effect.
Keep the final review short enough to use: what changed, what passed, what failed, how failures were resolved and what remains under observation. A CDN migration succeeds when delivery becomes dependable and the team can demonstrate it. Search visibility benefits from that foundation, but a CDN is not a guaranteed ranking improvement.
Frequently asked questions
Can moving to a CDN increase Googlebot traffic?
It can. Google explains that faster, stable responses may support increased crawling when demand exists. Check verified crawler activity and origin load together; neither a surge nor a fixed increase is guaranteed.
Does a CDN automatically cache HTML pages?
No. Cache eligibility depends on the provider and configuration. Cloudflare does not cache HTML or JSON by default. Inspect representative responses and their cache status before assuming page requests are offloaded from the origin.
How can we verify that a request is really Googlebot?
Use Google's published crawler IP ranges or its documented reverse-DNS check followed by a forward lookup that returns the original address. Do not trust the user-agent string alone, and select the correct crawler or fetcher category.
Should we use 429 or 503 responses during a migration?
Only when a temporary emergency measure is needed to relieve critical load, with an owner and a recovery check. Google's guidance describes short use and warns that persistent errors can harm visibility. Healthy migration traffic does not require deliberate errors.
What should a small business check first?
Start with the public home page, main service pages, essential assets, robots.txt and the enquiry journey. Confirm correct responses and legitimate crawler access. Expand the investigation when evidence shows an availability or crawling problem.
Sources and References
- Google Search Central: Moving to a CDN without overwhelming your origin, 11 November 2026.
- Google Crawling Infrastructure: Optimize your crawl budget.
- Cloudflare: Default cache behavior.
- Cloudflare: Cache responses.
- Google: Verify requests from crawlers and fetchers.
- Google: Reduce the Google crawl rate.
- Google Search Status Dashboard: November 2026 core update.


