Technical SEO · 10 min read

CDN Migration SEO: Protect Googlebot and Your Origin

Plan a CDN cutover with reliable crawler access, controlled origin load and clear recovery checks.

Two professionals reviewing a CDN cutover beside a laptop and server room

A CDN migration changes the route between your website and its visitors. It also changes the route used by search crawlers. A fast home page after the switch is encouraging, but it does not prove that product pages, older articles and essential resources remain available under sustained traffic.

On 11 November 2026, Google published guidance on CDN migrations and crawling. Faster responses can increase crawling when demand exists, while requests for uncached resources still reach the origin server. A second risk appears when teams respond to load by activating blanket bot challenges: legitimate crawlers can lose access.

For founders, ecommerce teams and technical leads, the useful question is whether the migration preserves reliable delivery across the whole site. This guide turns that question into a practical acceptance plan covering cache behaviour, crawler identity, security rules and recovery.

Understand the two systems you are changing

The origin is the application or server that produces your content. The CDN is an additional delivery layer that can serve eligible cached responses closer to visitors. Keep separate records of what happens at each layer. An edge response may look healthy while the application behind it is processing expensive requests.

Google's crawl-budget documentation separates capacity from demand. A responsive server can support more crawling, but that does not guarantee that Google will request more pages. Crawling also does not guarantee indexing. The advanced budget guidance mainly concerns large or rapidly changing sites; a small service business usually needs a proportionate availability check.

Define the scope of your project before testing. A CDN cutover with unchanged URLs is different from a simultaneous redesign, domain change or catalogue restructuring. Combining those changes makes a later regression harder to diagnose. Where practical, give each release its own evidence and rollback decision.

Confirm what the CDN actually caches

Moving traffic through a CDN does not automatically put every page in its cache. Cloudflare's default-cache documentation, for example, says HTML and JSON are not cached by default, while robots.txt is. Your provider, plan, application and rules determine the behaviour you will actually receive.

Build a small URL matrix before launch. Include a home page, important category, product or service page, older content page, image, script, stylesheet, robots.txt and sitemap. Add account, basket and checkout routes where applicable. For each, record whether caching is intended, whether the response varies by visitor, and who owns the decision.

Inspect repeated GET requests rather than relying on a browser reload alone. Record the status, content type, cache header and returned body. In Cloudflare's response terminology, HIT means the resource was found in cache; MISS means an eligible response was absent and came from the origin. DYNAMIC means the request was not eligible, while BYPASS reflects a response that was ultimately not cacheable.

Those distinctions prevent a common diagnostic mistake: repeatedly requesting a deliberately uncached page will not turn it into a useful cache hit. Conversely, a hit on a personalised response is a reason to investigate isolation, not celebrate a better ratio. Test signed-in and signed-out journeys separately, using safe test accounts.

Review exception rules before increasing cache coverage. Public product information may have different freshness requirements from images, while account details and basket state need correct user separation. Treat a broad cache rule as an application change with functional tests and a documented reversal path.

Establish a baseline the whole team can use

Choose a representative pre-migration period that includes normal trading patterns. For an India-based business with international customers, make the timezone explicit and compare equivalent hours. A quiet local morning may overlap with a busy overseas market. Record campaigns and scheduled imports that could explain unrelated traffic changes.

Keep the baseline focused on decisions:

  • Origin health: request volume, response time, application errors and the resource limits your hosting team actually monitors.
  • Delivery: edge and origin status codes, cache outcomes and examples of complete responses for important URL groups.
  • Crawler access: verified crawler requests, requested paths, challenge events and unsuccessful fetches.
  • Customer tasks: a working enquiry, product selection or checkout journey, with expected confirmation behaviour.

A technical SEO audit is useful here when it produces a concrete acceptance set: which URLs and signals must survive the cutover, who checks them, and what constitutes a release-stopping failure. A list of generic recommendations is less useful during an incident.

Assign an incident lead and separate application, security and search owners. Set thresholds from your own capacity and service requirements. There is no universal safe number of crawler requests per second, cache-hit percentage or migration duration that this guidance can supply.

Engineer inspecting network equipment beside an origin server cabinet
Illustrative scene: migration checks should cover both edge delivery and the origin application.

Protect crawler access without weakening security

Separate a request's claimed identity from its verified identity. An HTTP client can supply a Googlebot user-agent string; that label alone is not an access credential. Google documents verification through published IP ranges or reverse and forward DNS checks. Different crawler and fetcher categories use different lists.

Ask your security owner to document how the CDN identifies trusted crawlers and which exact rules they bypass. The desired outcome is access to the public content those crawlers need. It should not become a blanket exemption for every request claiming to be a bot, or permission to enter authenticated areas.

Test a legitimate public URL, a protected account route and a known unwanted request pattern. Save the matching security event or rule identifier where available. This makes it possible to show that a narrow exception fixed access while the relevant protections remained active.

Remember that a successful manual browser visit tests your own session and location. It does not establish what a crawler received elsewhere. Pair public fetch tests with actual verified crawler logs. When those disagree, investigate the delivery route, rule match and response body before changing content or indexing settings.

Run the cutover as a controlled release

Prepare a bounded warming set

Google recommends controlled cache warming for important and frequently crawled URLs. Turn that recommendation into a reviewed list with an owner and a rate limit. Include commercially important routes and the resources they need, then test whether those responses are eligible for caching.

Keep warming within a load level the origin can sustain. It is easy to create the very pressure you are trying to avoid by launching an unrestricted fetch of a large catalogue. Warming a sample is an operational preparation step, not proof that every edge location or URL is ready.

Preserve the content contract

Compare the public response before and after routing changes. Check the main content, canonical destination, indexing directives, redirects and structured data on the acceptance set. Also confirm that robots.txt and the sitemap contain the intended current version. Keep a saved response for any discrepancy so developers can reproduce it.

For a commerce site, run product discovery, variant selection and checkout checks as part of the same release. ecommerce SEO support should connect crawl access with usable catalogue pages: a technically reachable URL is insufficient when prices, availability or customer actions are incorrect.

Watch both sides of the connection

Keep origin and CDN observations together during the rollout. Record the time of each configuration change and avoid changing several security and cache settings at once. If a problem appears, a short sequence of isolated changes is much easier to reverse and explain.

Diagnose the symptom before choosing a remedy

Use the following as a triage sequence, then confirm the cause in your own logs. The patterns are practical examples rather than a prediction that every migration will encounter them.

  • Healthy pages, rising origin work: inspect which URL groups are reaching the application and whether their cache policy matches the plan. Check for unexpected query variations or intentionally dynamic routes before expanding capacity or caching.
  • Challenge pages or access denials: identify the security rule and verify the affected crawler. Repair the narrow rule responsible, then repeat the failed request and check subsequent crawler access.
  • Old content after an update: compare origin and public responses, identify the relevant cache entry or variant, and use the smallest appropriate invalidation. Retest content and headers together.
  • Failures concentrated in customer journeys: review session and application behaviour. Pause rollout expansion if a customer cannot complete the critical task, even when the public landing page is fast.

A helpful incident record contains a timestamp, affected URL group, observed response, suspected layer, corrective action and verification result. Keep the original evidence as well as the successful retest. This gives the next shift a usable handover and prevents an unexplained setting from becoming permanent.

Use emergency crawl reduction sparingly

If crawler traffic is causing critical load, Google's emergency crawl-rate guidance documents temporary 500, 503 or 429 responses. Significant errors can reduce crawling across the hostname, including otherwise healthy URLs. With 503 or 429, a Retry-After header can indicate when to retry.

This is an emergency control with consequences: fewer pages may be discovered or refreshed, and repeated errors over multiple days can lead to removal from Google's index. Google's guidance warns against maintaining it beyond a short period, such as a few hours or one to two days.

Before applying an emergency rule, name its owner, record the reason and set a review time. Once capacity recovers, remove the temporary condition and verify ordinary responses. Check that an intermediate cache is not continuing to serve the incident response after the application has recovered.

Do not treat a successful traffic reduction as the incident's only outcome. The release is stable when users can complete their tasks, necessary crawlers can fetch public content and the origin operates within agreed limits. Capture all three in the closure note.

Measure migration quality separately from rankings

For the immediate review, report availability, response correctness, crawler access and customer-task completion. Then follow indexing and organic performance over an appropriate comparison period. Annotate the migration date and other major events rather than assigning every movement to the CDN.

This distinction is especially relevant now. Google's November 2026 core-update notice records a global rollout beginning on 10 November and says it may take up to two weeks. A migration during that period needs an explicit measurement caveat; a ranking movement alone cannot isolate its effect.

Keep the final review short enough to use: what changed, what passed, what failed, how failures were resolved and what remains under observation. A CDN migration succeeds when delivery becomes dependable and the team can demonstrate it. Search visibility benefits from that foundation, but a CDN is not a guaranteed ranking improvement.

Frequently asked questions

Can moving to a CDN increase Googlebot traffic?

It can. Google explains that faster, stable responses may support increased crawling when demand exists. Check verified crawler activity and origin load together; neither a surge nor a fixed increase is guaranteed.

Does a CDN automatically cache HTML pages?

No. Cache eligibility depends on the provider and configuration. Cloudflare does not cache HTML or JSON by default. Inspect representative responses and their cache status before assuming page requests are offloaded from the origin.

How can we verify that a request is really Googlebot?

Use Google's published crawler IP ranges or its documented reverse-DNS check followed by a forward lookup that returns the original address. Do not trust the user-agent string alone, and select the correct crawler or fetcher category.

Should we use 429 or 503 responses during a migration?

Only when a temporary emergency measure is needed to relieve critical load, with an owner and a recovery check. Google's guidance describes short use and warns that persistent errors can harm visibility. Healthy migration traffic does not require deliberate errors.

What should a small business check first?

Start with the public home page, main service pages, essential assets, robots.txt and the enquiry journey. Confirm correct responses and legitimate crawler access. Expand the investigation when evidence shows an availability or crawling problem.

Sources and References

Continue reading

Related SEO guides

A beam crosses four glass diagnostic gates, with an amber disturbance highlighting a weak link.

Search strategy · 13 min read

SEO AEO GEO and SXO in 2026

Find the weak link in your search journey. Audit SEO, AEO, GEO and SXO with practical checks, current AI reporting and evidence-led priorities.

Indian publisher growth lead and SEO strategist reviewing an audience preference strategy

AI Search & Visibility · 11 min read

Google Preferred Sources: Publisher Action Plan

Learn how Google Preferred Sources works, who is eligible, where to place the publisher button, what to measure, and which ranking claims to avoid.

Indian cybersecurity lead and SEO strategist auditing a website for AI agent manipulation risks

Technical SEO · 11 min read

Prompt Injection: Protect AI Agents and SEO

Learn how prompt injection reaches AI agents through web content, why hidden instructions create SEO risk, and how to audit, contain and monitor exposure.

Ready to prioritize search growth?

Get a clear SEO roadmap before you spend another month guessing.

Share your site, target market, and biggest search challenge. We will map the first opportunities and the fixes that should happen first.

Request an SEO review