Blocking GPTBot stops OpenAI from using your pages for model training. Blocking OAI-SearchBot removes your pages from ChatGPT's live search index. Those are different outcomes, and most teams choose the wrong one by accident. AI crawlers now run split user-agents for training, real-time retrieval, and search indexing, and your CMS or CDN defaults rarely match what you actually want to allow. Parse tracks AI visibility across ChatGPT, Google AI Overviews, and Perplexity, which makes crawler-level mis-configurations visible as asymmetric citation coverage rather than sitewide declines.
- OpenAI, Anthropic, and Perplexity each run separate user-agents for training, on-demand fetches, and search indexing. Blocking the wrong one removes you from AI answers instead of training corpora.
Google-Extendedcontrols Gemini and Vertex AI training only. It does not affect Google Search or AI Overviews ranking.- robots.txt is a policy signal, not access control. Stealth crawlers ignore it, so pair directives with WAF rules when enforcement actually matters.
- AI crawlers do not execute JavaScript. Server-side rendering is a prerequisite before any
Allowrule pays off. - The fastest 15-minute audit:
curl -A "GPTBot" https://yourdomain.com/robots.txtagainst each crawler and compare what production serves to what your repo claims.
robots.txt is a visibility lever, not a privacy lever
Treat robots.txt as a policy signal, not access control. Well-behaved crawlers honor it. Stealth and unverified crawlers ignore it. Its real leverage on AI visibility is telling each crawler which job it is allowed to do on your site, and most teams have never made that decision deliberately.
The market has started to split. Q1 2026 data from Cloudflare's network shows ClaudeBot's share of Disallow rules rising from 9.6% to 10.1% in three months, while PerplexityBot now appears in Allow rules more often than Disallow, because sites have begun distinguishing training crawlers from search crawlers. If your robots.txt was last updated before this split existed, it is almost certainly misconfigured for the goals you would set today.
OpenAI runs three crawlers. Block the wrong one and you vanish from ChatGPT Search.
OpenAI separates three jobs across three user-agents. GPTBot fetches pages as candidate training data for future model releases. OAI-SearchBot builds the index that answers live queries inside ChatGPT Search. ChatGPT-User acts on behalf of a specific user who asked ChatGPT to visit a URL, and OpenAI's own documentation notes that because those visits are user-initiated, robots.txt "may not apply."
The common mistake: a blanket User-agent: GPTBot / Disallow: / feels like a reasonable AI opt-out, but leaves OAI-SearchBot untouched and still crawling. The inverse is worse, blocking OAI-SearchBot while allowing GPTBot gives OpenAI your content for training while removing your pages from ChatGPT Search answers, which is usually the opposite of what a brand wants.
The crawler reference table
Treat this as the working set you need to make a deliberate decision on. These are the user-agent tokens site owners encounter most often in 2026.
| User-agent | Operator | Job | What blocking removes |
|---|---|---|---|
GPTBot | OpenAI | Training-data collection for future models | Your pages from future GPT training corpora |
OAI-SearchBot | OpenAI | Index for ChatGPT Search | Your pages from ChatGPT Search answers |
ChatGPT-User | OpenAI | On-demand fetch when a user asks ChatGPT to visit a URL | May be ignored, user-initiated, robots.txt "may not apply" |
ClaudeBot | Anthropic | Training-data collection | Your pages from Claude training corpora |
Claude-User | Anthropic | On-demand fetch during a Claude conversation | Claude's ability to read the page for an asking user |
Claude-SearchBot | Anthropic | Index for Claude's search feature | Your pages from Claude search answers |
PerplexityBot | Perplexity | Search index for Perplexity | Your pages from Perplexity answers |
Perplexity-User | Perplexity | User-initiated fetch | Ignored for directly user-prompted visits |
Google-Extended | Signals Gemini and Vertex AI training opt-out only | AI training use; does not affect Google Search | |
CCBot | Common Crawl | Open crawl used by many downstream LLMs | Your pages from Common Crawl corpora used by multiple labs |
Anthropic's three-bot model and its relatively firm robots.txt stance
Anthropic now publishes a three-bot split that mirrors OpenAI's. ClaudeBot collects training data. Claude-User fetches pages in-conversation when a user asks Claude to visit a URL. Claude-SearchBot crawls for the search feature inside Claude.
The notable policy difference: Anthropic states that all three bots honor robots.txt, including the non-standard Crawl-delay extension. That is a firmer commitment than OpenAI offers for ChatGPT-User, whose documentation explicitly warns robots.txt "may not apply" on user-initiated fetches. In practice, if you want to signal "do not train, but do answer live questions about my site," you can Disallow ClaudeBot and Allow Claude-SearchBot and Claude-User with reasonable confidence that Anthropic will comply, while treating the equivalent OpenAI pairing as best-effort.
Perplexity, Google-Extended, and the stealth-crawler problem
Perplexity publishes PerplexityBot for indexing and Perplexity-User for user-initiated fetches. Its behavior has been contested: in August 2025, Cloudflare documented that when its declared crawlers were blocked via robots.txt or firewall rules, Perplexity continued accessing the same content from rotating IPs under generic browser user-agents. Cloudflare delisted Perplexity from its verified-bot list as a result. This does not change what you should put in robots.txt, you still declare intent there, but it does change how you enforce it. If you genuinely need Perplexity traffic blocked, pair the directive with a WAF rule.
Google-Extended is not a separate crawler at all. It is a robots.txt control token that tells Google whether your content can train Gemini and Vertex AI generative APIs. It has no effect on Google Search or on AI Overviews ranking signals.
If you want to see which sources shape AI answers about your brand, run a free brand check — it takes a minute.
The default configuration for brands that want AI visibility
For the vast majority of brands, where the goal is to appear in AI answers, not to opt out of AI, the correct default is explicit and open. Do not rely on the absence of directives. Many CDNs, hosting providers, and CMS platforms now inject restrictive defaults during setup. A 2026 analysis of Anthropic bots across commonly used hosts found ClaudeBot implicitly blocked on a meaningful share of sites whose owners had no intention of blocking it.
A minimum viable allow-list for a visibility-oriented brand:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://example.com/sitemap.xml
Reviewing this block against your current file is usually a 15-minute audit that removes an invisible drag on AI citation coverage.
The "training opt-out" configuration: block training, keep search
Some brands, most often publishers, licensed-content businesses, and legal or regulated industries, have a defensible reason to block training while keeping AI search visibility. The split is supported explicitly by OpenAI and Anthropic, and has become the fastest-growing configuration pattern on Cloudflare's network.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
This configuration signals: do not use us to train future models, but do retrieve and cite us when users ask live questions. It is the operational meaning of "opt out of AI training" for most publishers.
What robots.txt cannot do
Three limits matter, and pretending otherwise causes expensive mistakes.
First, robots.txt is not access control. Stealth crawlers, scrapers, and fine-tuning pipelines that source from third-party datasets will ignore it. If you need enforcement, you need WAF rules, bot-management services, or authenticated gates.
Second, AI crawlers do not execute JavaScript. An analysis of over 500 million GPTBot fetches reported by Vercel found no evidence of JavaScript rendering; the same pattern holds across OpenAI, Anthropic, and Perplexity crawlers. If your page relies on client-side rendering to hydrate content, the crawler sees an empty shell regardless of what your robots.txt allows. Server-side rendering or pre-rendering is a prerequisite before allow-lists matter.
Third, Allow: / does not guarantee citation. It only removes a block. What happens after the crawler fetches the page is a separate problem. Ghost citations cover the case where your content is read but your brand is still not named, and AI citation gap analysis covers how to quantify the shortfall against competitors.
How to validate your configuration is actually working
Intent in robots.txt is cheap. Verification is what most teams skip.
- Fetch your live file from production (not staging) with
curl -A "GPTBot" https://yourdomain.com/robots.txtand repeat for each crawler you care about. Compare what is actually served to what your source-of-truth config says. CDN overrides, WAF rules, and Cloudflare's per-bot defaults frequently diverge from the file in your repo. - Check server logs for each user-agent over a 30-day window. A correctly allowed crawler that never appears is often being blocked upstream by a bot-management service.
- Run an AI search audit for your target prompts. If you are cited on some platforms but missing on others in the same query set, crawler access is a plausible root cause. How ChatGPT decides which brands to recommend explains the retrieval and recommendation split that makes this diagnosis concrete.
- Re-check after any platform migration, CDN change, or security-policy update. Those are the moments when robots.txt silently regresses.
FAQ
Does allowing GPTBot expose my content to training?
Yes. GPTBot is OpenAI's training-data crawler; allowing it means pages it fetches become candidates for future model training. If that outcome is unacceptable for legal, licensing, or competitive reasons, Disallow GPTBot specifically while leaving OAI-SearchBot and ChatGPT-User allowed. This preserves your ability to appear in ChatGPT Search answers without contributing to model weights.
Will blocking OAI-SearchBot hide my site from ChatGPT?
It will remove your pages from ChatGPT's live search index, which is how ChatGPT answers queries that require the web. You may still be referenced from what the model already learned during training, but you will not appear in fresh, web-retrieved citations for current queries. For a brand that cares about AI visibility, blocking OAI-SearchBot is usually the wrong choice.
Is Google-Extended the same as blocking Google Search?
No. Google-Extended only controls whether your content can train Gemini and Vertex AI. Googlebot is a separate user-agent that handles Google Search indexing, including the signals that feed AI Overviews ranking. You can Disallow: Google-Extended and keep full Google Search coverage. Conflating the two is the most common robots.txt mistake documented in recent audits.
Can a user-initiated ChatGPT or Claude fetch ignore robots.txt?
Sometimes. OpenAI's documentation states that ChatGPT-User visits, being user-initiated, "may not apply" robots.txt rules. Anthropic is firmer, Claude-User is documented to honor robots.txt. If you need strict enforcement of a block against on-demand fetches, treat robots.txt as a signal and add server-side or WAF controls.
How often should we review our robots.txt for AI crawlers?
Quarterly at minimum, plus after any CDN, hosting, or security-layer change. Crawler names, policies, and stated behaviors are still shifting, Anthropic formalized its three-bot split in early 2026, OpenAI has revised ChatGPT-User policy language multiple times, and Perplexity's crawler status changed mid-2025. A file that was correct 12 months ago is unlikely to match your current intent.
:::
Parse surfaces which AI platforms are citing your pages today, which makes a misconfigured robots.txt visible as asymmetric coverage across ChatGPT, Google AI Overviews, Perplexity, and Claude. If the gaps look crawler-shaped rather than content-shaped, see your citation gaps to diagnose where access, not content, is the blocker.