AI crawler policies should distinguish training access, search indexing and user-requested retrieval. Check the provider’s current documentation for each user-agent and verify your actual server responses. A robots.txt rule describes an access preference; citation coverage alone cannot diagnose whether a crawler was blocked.
- OpenAI, Anthropic, and Perplexity each run separate user-agents for training, on-demand fetches, and search indexing. Blocking the wrong one removes you from AI answers instead of training corpora.
Google-Extendedcontrols Gemini and Vertex AI training only. It does not affect Google Search or AI Overviews ranking.- robots.txt is a policy signal, not access control. Stealth crawlers ignore it, so pair directives with WAF rules when enforcement actually matters.
- AI crawlers do not execute JavaScript. Server-side rendering is a prerequisite before any
Allowrule pays off. - The fastest 15-minute audit:
curl -A "GPTBot" https://yourdomain.com/robots.txtagainst each crawler and compare what production serves to what your repo claims.
Verify the response beyond robots.txt
A permissive robots.txt rule does not prove a crawler can read the article. Test the exact public URL, then check server or CDN logs for the crawler's real requests. A challenge page, login response or empty app shell can block useful retrieval even when the status is 200.
| Check | Evidence to retain |
|---|---|
| Applicable robots.txt group | The rule matching the user-agent and path |
| Network access | A successful request from a verified crawler IP |
| Page response | Article text in the returned HTML, not a challenge |
| Search eligibility | Indexing and snippet controls for the relevant engine |
| Citation outcome | Repeated answers containing the actual page URL |
OpenAI's crawler documentation distinguishes search, training and user-requested visits. Allowing OAI-SearchBot does not require allowing GPTBot. Blocking search crawling can still leave navigational links visible. The documentation also lists OAI-AdsBot for submitted ad landing pages; that is a separate purpose from organic search.
For Google AI Overviews and AI Mode, use the Google Search controls. Google-Extended is not the switch for inclusion in those Search features.
robots.txt is a visibility lever, not a privacy lever
Treat robots.txt as a policy signal, not access control. Well-behaved crawlers honor it. Stealth and unverified crawlers ignore it. Its real leverage on AI visibility is telling each crawler which job it is allowed to do on your site, and most teams have never made that decision deliberately.
The market has started to split. Q1 2026 data from Cloudflare's network shows ClaudeBot's share of Disallow rules rising from 9.6% to 10.1% in three months, while PerplexityBot now appears in Allow rules more often than Disallow, because sites have begun distinguishing training crawlers from search crawlers. If your robots.txt was last updated before this split existed, it is almost certainly misconfigured for the goals you would set today.
Separate OpenAI search crawling from training and user visits
For organic discovery, distinguish these three OpenAI user-agents. GPTBot fetches pages as candidate training data for future model releases. OAI-SearchBot builds the index that answers live queries inside ChatGPT Search. ChatGPT-User acts on behalf of a specific user who asked ChatGPT to visit a URL, and OpenAI's own documentation notes that because those visits are user-initiated, robots.txt "may not apply."
The common mistake: a blanket User-agent: GPTBot / Disallow: / feels like a reasonable AI opt-out, but leaves OAI-SearchBot untouched and still crawling. The inverse is worse, blocking OAI-SearchBot while allowing GPTBot gives OpenAI your content for training while removing your pages from ChatGPT Search answers, which is usually the opposite of what a brand wants.
The crawler reference table
Use this list to decide which crawlers to allow or block in 2026.
| User-agent | Operator | Job | What blocking removes |
|---|---|---|---|
GPTBot | OpenAI | Training-data collection for future models | Your pages from future GPT training corpora |
OAI-SearchBot | OpenAI | Index for ChatGPT Search | Your pages from ChatGPT Search answers |
ChatGPT-User | OpenAI | On-demand fetch when a user asks ChatGPT to visit a URL | May be ignored, user-initiated, robots.txt "may not apply" |
ClaudeBot | Anthropic | Training-data collection | Your pages from Claude training corpora |
Claude-User | Anthropic | On-demand fetch during a Claude conversation | Claude's ability to read the page for an asking user |
Claude-SearchBot | Anthropic | Index for Claude's search feature | Your pages from Claude search answers |
PerplexityBot | Perplexity | Search index for Perplexity | Your pages from Perplexity answers |
Perplexity-User | Perplexity | User-initiated fetch | Ignored for directly user-prompted visits |
Google-Extended | Controls certain Gemini and Vertex AI uses | Covered training and grounding uses; not Google Search | |
CCBot | Common Crawl | Open crawl used by many downstream LLMs | Your pages from Common Crawl corpora used by multiple labs |
If you want to see which sources shape AI answers about your brand, run a free brand check. It takes a minute.
Anthropic's three-bot model and its relatively firm robots.txt stance
Anthropic now publishes a three-bot split that mirrors OpenAI's. ClaudeBot collects training data. Claude-User fetches pages in-conversation when a user asks Claude to visit a URL. Claude-SearchBot crawls for the search feature inside Claude.
The notable policy difference: Anthropic states that all three bots honor robots.txt, including the non-standard Crawl-delay extension. That is a firmer commitment than OpenAI offers for ChatGPT-User, whose documentation explicitly warns robots.txt "may not apply" on user-initiated fetches. In practice, if you want to signal "do not train, but do answer live questions about my site," you can Disallow ClaudeBot and Allow Claude-SearchBot and Claude-User with reasonable confidence that Anthropic will comply, while treating the equivalent OpenAI pairing as best-effort.
Perplexity, Google-Extended, and the stealth-crawler problem
Perplexity publishes PerplexityBot for indexing and Perplexity-User for user-initiated fetches. Its behavior has been contested: in August 2025, Cloudflare documented that when its declared crawlers were blocked via robots.txt or firewall rules, Perplexity continued accessing the same content from rotating IPs under generic browser user-agents. Cloudflare delisted Perplexity from its verified-bot list as a result. This does not change what you should put in robots.txt, you still declare intent there, but it does change how you enforce it. If you genuinely need Perplexity traffic blocked, pair the directive with a WAF rule.
Google-Extended is not a separate crawler at all. It is a robots.txt control token that tells Google whether your content can train Gemini and Vertex AI generative APIs. It has no effect on Google Search or on AI Overviews ranking signals.
The default configuration for brands that want AI visibility
For the vast majority of brands, where the goal is to appear in AI answers, not to opt out of AI, the correct default is explicit and open. Do not rely on the absence of directives. Many CDNs, hosting providers, and CMS platforms now inject restrictive defaults during setup. A 2026 analysis of Anthropic bots across commonly used hosts found ClaudeBot implicitly blocked on a meaningful share of sites whose owners had no intention of blocking it.
A minimum viable allow-list for a visibility-oriented brand:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://example.com/sitemap.xml
Reviewing this block against your current file is usually a 15-minute audit that removes an invisible drag on AI citation coverage.
The "training opt-out" configuration: block training, keep search
Some brands, most often publishers, licensed-content businesses, and legal or regulated industries, have a defensible reason to block training while keeping AI search visibility. The split is supported explicitly by OpenAI and Anthropic, and has become the fastest-growing configuration pattern on Cloudflare's network.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
This configuration signals: do not use us to train future models, but do retrieve and cite us when users ask live questions. It is the operational meaning of "opt out of AI training" for most publishers.
What robots.txt cannot do
Three limits matter, and pretending otherwise causes expensive mistakes.
First, robots.txt is not access control. Stealth crawlers, scrapers, and fine-tuning pipelines that source from third-party datasets will ignore it. If you need enforcement, you need WAF rules, bot-management services, or authenticated gates.
Second, AI crawlers do not execute JavaScript. An analysis of over 500 million GPTBot fetches reported by Vercel found no evidence of JavaScript rendering; the same pattern holds across OpenAI, Anthropic, and Perplexity crawlers. If your page relies on client-side rendering to hydrate content, the crawler sees an empty shell regardless of what your robots.txt allows. Server-side rendering or pre-rendering is a prerequisite before allow-lists matter.
Third, Allow: / does not guarantee citation. It only removes a block. What happens after the crawler fetches the page is a separate problem. Ghost citations cover the case where your content is read but your brand is still not named, and AI citation gap analysis covers how to quantify the shortfall against competitors.
How to validate your configuration is actually working
Intent in robots.txt is cheap. Verification is what most teams skip.
- Fetch your live file from production (not staging) with
curl -A "GPTBot" https://yourdomain.com/robots.txtand repeat for each crawler you care about. Compare what is actually served to what your source-of-truth config says. CDN overrides, WAF rules, and Cloudflare's per-bot defaults frequently diverge from the file in your repo. - Check server logs for each user-agent over a 30-day window. A correctly allowed crawler that never appears is often being blocked upstream by a bot-management service.
- Run an AI search audit for your target prompts. If you are cited on some platforms but missing on others in the same query set, crawler access is a plausible root cause. How ChatGPT decides which brands to recommend explains the retrieval and recommendation split that makes this diagnosis concrete.
- Re-check after any platform migration, CDN change, or security-policy update. Those are the moments when robots.txt silently regresses.
Parse surfaces which AI platforms are citing your pages today, which makes a misconfigured robots.txt visible as asymmetric coverage across ChatGPT, Google AI Overviews, Perplexity, and Claude. If the gaps look crawler-shaped rather than content-shaped, see your citation gaps to diagnose where access, not content, is the blocker.