AI crawlers no longer have one simple purpose. One bot may build a search index, another may retrieve a page for a live answer, and another may collect material for model training. Treating all of them alike can create two opposite failures: giving away more access than the business intended, or blocking the discovery channels that still produce qualified traffic and citations.
The practical answer is not a universal “block AI” switch. It is a documented access policy that separates search discovery, real-time AI answers, model improvement and user-triggered visits. That policy then needs to be expressed through the controls each operator actually supports, tested at the server, and monitored in logs.
This guide explains how to build that policy without confusing robots.txt, indexing controls and technical enforcement. It focuses on current documentation from Google, OpenAI, Cloudflare and the Internet Engineering Task Force rather than promises about rankings or AI citations.
Why AI crawler control is now a search decision
Search visibility and AI use are related, but they are not identical. Google states in its documentation for AI features and websites that Googlebot controls access for Search, including AI features within Search. The same page directs publishers to Google-Extended for limiting training and grounding in certain other Google systems.
That distinction matters. Blocking Googlebot can remove the underlying access needed for ordinary Google Search as well as its AI features. Blocking Google-Extended has a different scope. Google's common crawler reference, updated on 14 July 2026, says Google-Extended is a standalone control token for training future Gemini models and grounding in Gemini Apps and Vertex AI. It also says this control does not affect inclusion or ranking in Google Search.
OpenAI also publishes separate machine-readable IP-range records for OAI-SearchBot and GPTBot. Its official response to the UK copyright consultation notes that robots.txt requires publishers either to name a particular crawler or set blanket rules for unknown crawlers regardless of purpose. Separate tokens and verifiable ranges make a purpose-based decision possible, but the business still has to define the rule.
The business question is therefore more precise than “Do we allow AI?” It is: which content may be discovered, quoted, used at request time, used for training, or not fetched at all?
Start with a content-use matrix
Do not write bot rules before agreeing on content value and risk. Build a small matrix with content groups as rows and uses as columns. The columns should cover at least conventional search, AI-answer retrieval, model training and user-triggered access.
Public acquisition content
Service pages, product categories, location pages, public research and editorial guides normally exist to be discovered. Blocking search-oriented crawlers from these pages can conflict with their commercial purpose. The better question is whether training access should follow the same policy as discovery access. It does not have to.
Licensed or subscriber-funded material
A publisher may want headlines and short public summaries discoverable while reserving full reports, datasets or paid analysis. Access control should follow the real entitlement boundary. A login, signed URL or server-side authorisation is stronger than a crawler request in robots.txt.
Private, regulated or operational content
Customer records, draft contracts, internal documentation and non-public price files should not depend on crawler compliance. If a URL contains sensitive information, protect it through authentication and proper server permissions. Hiding it from a sitemap or adding a bot rule does not make it private.
For every row, name the owner, commercial objective, permitted uses, restricted uses and review date. This turns a technical file into an accountable business decision rather than an accumulation of copied rules.
Understand what robots.txt can and cannot do
The Robots Exclusion Protocol in RFC 9309 standardises how crawlers find and interpret groups of user agents, Allow rules and Disallow rules. It also makes an essential limitation explicit: these rules are not access authorisation.
In other words, robots.txt communicates a request to compliant automated clients. It does not prevent a browser, an unidentified scraper or a bot that ignores the protocol from requesting a public URL. It is useful for declared crawler governance, but it is not a substitute for authentication, a web application firewall or rate limiting.
Cloudflare makes the same distinction in its current managed robots.txt documentation. It describes compliance as voluntary and recommends enforcement through AI Crawl Control when a publisher needs to stop access rather than merely express a preference.
This leads to a three-layer model:
- Preference: declare allowed and disallowed paths for named crawlers in
robots.txt. - Presentation and indexing: use page-level robots directives and HTTP headers to control indexing and snippets.
- Enforcement: use authentication, rate limits, verified-bot rules or edge security controls when access must actually be denied.
Each layer solves a different problem. Combining them without understanding the boundary is how discoverable pages become inaccessible or restricted pages remain exposed.
Do not confuse crawl blocking with noindex
A Disallow rule prevents a compliant crawler from fetching a path. A noindex instruction asks a search engine not to show the resource in results. These are not interchangeable.
Google's robots meta tag specification says page-level instructions can be followed only when crawlers are allowed to access the page. If Googlebot cannot fetch a URL because robots.txt blocks it, Google may not see the noindex tag or X-Robots-Tag response header. The URL can remain known through links or previous discovery even though its content is not crawled.
Use noindex for public pages that a search engine may fetch but should not list. Use an X-Robots-Tag for non-HTML resources such as PDFs when page markup is unavailable. Use nosnippet, max-snippet or data-nosnippet when the goal is to limit how much content appears in Google results rather than remove the page.
For genuinely confidential material, none of these is sufficient. Require authentication or remove public access. A crawler directive is not a security boundary.
Build a bot policy without breaking discovery
A useful policy begins with the operator's official documentation and a narrow test. Do not copy a generic block list from a forum: crawler names, purposes and supported controls change. Verify the current user-agent token, affected products and any published IP-verification method.
Separate Google Search from other Google uses
If public pages should remain eligible for Google discovery, avoid blocking Googlebot. A publisher that permits Search but restricts the uses covered by Google-Extended can express those choices separately:
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /This example is not a recommendation for every site. It illustrates purpose separation. Before implementation, review image, video, news and product-specific crawlers as well as contractual obligations.
Configure OpenAI crawler tokens separately
OpenAI's separate crawler records support operator-specific verification. A site can also address the tokens independently in robots.txt:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /This example illustrates separate treatment, not a universal recommendation about which token to allow. Confirm each crawler's current purpose in OpenAI's live documentation, verify traffic against the published ranges, and record the business decision and review date. No allow rule guarantees a citation, referral or other commercial result.
Use path-level rules only when the content model is clear
A mixed site may allow crawlers across public guides while disallowing licensed reports. Test the longest and most specific path matches, case sensitivity, URL encoding and subdomain scope. RFC 9309 places robots.txt at the top-level path of each service, so a rule on the main domain does not automatically govern every separate host.
Enforce restrictions when preference is not enough
When logs show non-compliance, high-volume extraction or spoofed identities, move beyond user-agent matching. Cloudflare's AI Crawl Control documentation, updated on 28 July 2026, describes per-crawler allow and block actions, robots.txt violation reporting and WAF-backed enforcement. It also warns that free-plan identification relies on user-agent strings, while deeper bot detection is available through additional controls.
The principle applies beyond one vendor. An enforcement design should consider verified IP ranges or reverse DNS where officially supported, request behaviour, rate, path sensitivity and false positives. A browser-like user agent alone is weak evidence of identity.
Start in observation mode where possible. Compare declared crawler identity with server logs, identify the pages requested and estimate bandwidth or origin load. Then test a rule on a limited path. Confirm that ordinary users, monitoring services and intended search crawlers remain unaffected before expanding it.
A structured technical SEO audit is useful before broad crawler changes because existing redirects, canonical rules, faceted URLs and edge configurations can alter the result. The safest policy is one that has been tested against the actual delivery stack, not only reviewed as text.
Measure the result with a 30-day control loop
Crawler governance is not finished when a file is deployed. Operators can change documentation, pages can move, and security tools can classify traffic differently. Use a recurring control loop.
Week 1: baseline and test
Export at least 28 days of server or CDN logs. Record requests by verified crawler, user-agent, path group, response code and bytes served. Capture current Google Search Console performance and any measurable AI referrals before changing rules. Validate the live /robots.txt response on every relevant host.
Week 2: deploy narrowly
Implement operator-specific rules for one content group. Request the file directly, use the operator's testing tools where available, and verify that the intended crawler receives the intended response. Check that no CMS, CDN or security product overwrites the file.
Weeks 3 and 4: watch technical and commercial effects
Monitor crawl activity, blocked requests, index coverage, search impressions, source referrals and origin load. Separate correlation from causation: a visibility change may coincide with a core update, publishing change or seasonality. Preserve the deployment date and rule diff so the team can investigate.
An AI visibility assessment can complement log evidence by checking whether important brand and service facts continue to appear accurately across answer surfaces. It should not replace technical measurement, because an observed answer does not reveal which crawler, index or retrieval path produced it.
Govern the policy as the business changes
Different businesses should reach different decisions. A local service company in Delhi NCR may value discoverability and referrals more than training restrictions on public service pages. A research publisher may permit indexing but restrict training access to licensed reports. An ecommerce team may keep product pages discoverable while protecting supplier feeds, account areas and unpublished launches through real access controls.
Assign one business owner and one technical owner. Review the matrix quarterly and whenever a crawler operator changes its documentation. Keep a versioned record of the rule, rationale, affected hosts, approval, test evidence and rollback method. Legal or licensing questions should be reviewed by qualified counsel; a technical directive alone does not settle contractual or copyright rights.
The most durable policy is not the most restrictive one. It is the one that accurately reflects the value of each content class, distinguishes crawler purposes, uses the right technical layer and can be verified after deployment.
Frequently asked questions
Will blocking AI training crawlers hurt Google rankings?
It depends on the crawler being blocked. Google states that Google-Extended does not affect inclusion or ranking in Google Search, while blocking Googlebot affects Search crawling. Verify each operator's current documentation instead of assuming every AI-related token has the same purpose.
Does robots.txt keep private content secure?
No. RFC 9309 says robots rules are not access authorisation. Use authentication, server permissions or another enforceable access control for confidential or paid material.
Can a page be blocked in robots.txt and also use noindex?
The tags can coexist, but blocking the crawl may prevent the search engine from seeing the noindex instruction. If removal from results is the goal, allow the relevant crawler to fetch the instruction, or use an appropriate removal and access-control process.
Should OAI-SearchBot and GPTBot have the same rule?
Not automatically. OpenAI publishes separate crawler records for the two tokens, and its copyright submission explains why purpose-blind blanket rules are limited. Verify their current roles in official documentation and decide separately based on the site's content and commercial objectives.
How often should crawler policies be reviewed?
Review them at least quarterly and after major platform, CDN, CMS or business-model changes. Recheck official bot documentation, live responses, server logs, indexing and referral outcomes.
Sources and References
- Google Search Central: AI features and your website, current documentation accessed 30 July 2026.
- Google Crawling Infrastructure: list of common crawlers, updated 14 July 2026.
- OpenAI: OAI-SearchBot published IP ranges, accessed 30 July 2026.
- OpenAI: GPTBot published IP ranges, accessed 30 July 2026.
- OpenAI: response to the UK copyright and AI consultation, February 2025.
- IETF RFC 9309: Robots Exclusion Protocol, September 2022.
- Cloudflare: managed robots.txt and Content Signals, current documentation accessed 30 July 2026.
- Cloudflare: Manage AI crawlers, updated 28 July 2026.
