Every AI engine that cites the web sends crawlers to read it, and most makers now run more than one. One finds pages for answers, one fetches a page when a person asks about it, and one collects training data. In a server log they look alike, but blocking each one costs something different.
This guide lists the crawlers each maker documents, what each is for, and a robots.txt that keeps you in AI answers while leaving the training question to you.
On this page
Three jobs, three kinds of crawler
Each maker splits its crawling by purpose, and the purpose is what matters for visibility:
- Search crawlers build the index an engine searches when it writes an answer. Block one and the engine has little of yours to cite.
- User fetchers load a page because a person asked about it, for example by pasting a link into a chat. Several makers say these may not follow robots.txt, since a person asked.
- Training crawlers collect text to train future models. Blocking them doesn't remove you from answers that search the web.
So "should I block AI crawlers?" is really three questions. Most sites that want to be recommended should answer the first two with yes and treat the third as a business decision.
The crawlers each maker documents
These are the tokens the makers publish themselves. Community lists name many more, with behaviour nobody has confirmed; it's safer to write rules for documented tokens than for guesses.
OpenAI
- OAI-SearchBot finds pages for
ChatGPT search answers. OpenAI says sites that block it won't appear in ChatGPT search answers, apart from navigational links, and that a change takes about a day. - ChatGPT-User fetches a page when a ChatGPT user's request needs it. OpenAI says robots.txt rules may not apply to it.
- GPTBot collects training data. It's independent of OAI-SearchBot: blocking it opts out of training only.
- OAI-AdsBot checks the landing pages of ads submitted to ChatGPT; it has nothing to do with organic answers.
Anthropic
- Claude-SearchBot crawls to improve the quality of
Claude search results. - Claude-User fetches a page when someone asks Claude a question that needs it.
- ClaudeBot collects training data.
Anthropic says all three respect robots.txt, including the non-standard Crawl-delay line.
Perplexity
- PerplexityBot builds the index
Perplexity answers from. Perplexity says it isn't used to train models, and blocking it makes your pages unlikely to appear. - Perplexity-User fetches pages for people's requests, and Perplexity says it generally ignores robots.txt.
- Googlebot crawls for Google Search, which
Google AI Overviews and
Google AI Mode answer from. A page Googlebot can't crawl can't be a supporting link there; blocking it also removes you from Search itself. - Google-Extended isn't a crawler but a robots.txt token. It decides whether Google may use your pages to train Gemini models and to ground answers in the
Gemini app. It doesn't affect Google Search.
That second point is the trap: many sites block Google-Extended to opt out of training and, without meaning to, drop out of Gemini's grounded answers too.
Meta
Meta documents three crawlers for Meta AI. Its assistant
Muse cites the web, so these are the ones that matter there:
- Meta-WebIndexer crawls to improve Meta AI's search results; Meta says allowing it helps cite and link your pages in its AI answers.
- Meta-ExternalAgent crawls for training and for indexing content directly.
- Meta-ExternalFetcher fetches pages for a user's request and, Meta says, may bypass robots.txt.
xAI
xAI documents no crawler for
Grok, so robots.txt has no Grok token to allow or block. Being mentioned on the pages Grok cites is the more reliable lever.
A robots.txt that keeps you in the answers
A starting point that lets every documented search and user crawler in and opts out of training, which you can drop if you're happy to be trained on:
# Search and user crawlers: let them read the site
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Meta-WebIndexer
Disallow: /account/
Allow: /
# Training only: your choice
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Meta-ExternalAgent
Disallow: /
# Everyone else
User-agent: *
Disallow: /account/
Two details make or break a file like this:
- A crawler obeys only the group that names it. Under the robots.txt standard (RFC 9309) a crawler follows the most specific group matching its name and ignores the
*group. If your*group keeps crawlers out of/account/, repeat that line in every named group, as above, or the named crawlers may go where you didn't intend. - Leave Googlebot and Google-Extended alone unless you mean it. Googlebot falls under the
*group here; Google-Extended isn't listed, so Gemini may still ground answers in your pages.
Check the layer above robots.txt
robots.txt is a request a crawler reads. A CDN or firewall in front of your site can turn a crawler away before it reads anything, and then your robots.txt doesn't matter.
Cloudflare is the common case. Since July 2025 it has blocked AI crawlers by default on new domains, and since July 2026 it sorts them into search, agent and training bots that can be allowed or blocked separately, on every plan. If your site sits behind Cloudflare or another CDN, open its bot settings and check that search crawlers are allowed there too.
Google adds one more switch. Search Console's "Search generative AI" control decides whether your site may appear in AI Overviews, AI Mode and Discover's generative AI features. Sites are included by default, and Google says a site must be included to be eligible; it's worth a look if someone on your team has visited that setting.
What blocking training crawlers costs
Answers that search the web don't depend on training crawlers. Answers that don't search lean on what a model learned in training, and some engines answer many questions without searching at all. Opting out of training doesn't remove you from search-based answers, but it does change what the next models learn about you.
No published study has measured that effect on recommendations, so treat it as a licensing and business decision, not a ranking one. Whatever you choose, choose it per crawler, with the trade-off in view.
How to check your site
- Open
yoursite.com/robots.txtand find the group each crawler above falls under, remembering that a named group replaces the*group. - Check your CDN's or firewall's bot settings for blocked search or user crawlers.
- Look for the crawlers in your server or CDN logs. OpenAI, Anthropic and Perplexity publish the IP ranges their crawlers use, so you can tell a real crawler from someone borrowing its name.
- Recheck after a redesign or a CDN change, which is when these rules most often break.
Questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data only. Whether ChatGPT's search answers can show your pages depends on OAI-SearchBot, which OpenAI documents as a separate crawler.
What is Google-Extended?
A robots.txt token Google reads, not a crawler. Blocking it opts out of training Gemini models and also keeps your pages out of grounded answers in the Gemini app. It doesn't affect Google Search, AI Overviews or AI Mode.
How long does a robots.txt change take to apply?
OpenAI says about a day for OAI-SearchBot, and Perplexity says up to 24 hours. Search engines also cache robots.txt, so allow a day or two before reading anything into the results.
Do AI crawlers obey robots.txt?
The search and training crawlers their makers document say they do. The user fetchers are different: OpenAI, Perplexity and Meta say theirs may not follow robots.txt, because a person asked for the page.
Sources
- OpenAI: overview of OpenAI crawlers developers.openai.com
- Anthropic: does Anthropic crawl data from the web? support.claude.com
- Perplexity: Perplexity crawlers docs.perplexity.ai
- Google: AI features and your website developers.google.com
- Google: common crawlers (Google-Extended) developers.google.com
- Meta: web crawlers developers.facebook.com
- RFC 9309: Robots Exclusion Protocol rfc-editor.org
- Cloudflare: AI crawlers blocked by default (July 2025) cloudflare.com
- Cloudflare: search, agent and training bot controls (July 2026) blog.cloudflare.com
- Google Search Console Help: Search generative AI control support.google.com



